Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Differentiable Neural Computers

deepmind.com

Post reply on HN
Print view View original

71–77 of 77 posts

Re: Differentiable Neural Computers

#71

Post by JeffreyKaine »

tvural wrote: post #69

The idea of using neural networks to do what humans can already write code to do seems a bit wrong-headed. Why would you take a system that's human-readable, fast, and easy to edit, and make it slow, opaque, and very hard to edit? The big wins for ml have all been things that people couldn't write code to do, like image recognition.

I think they just want to teach the system to crawl before it can walk, run then eventually fly. Doing something that would be easy for a human to code makes it easy for a human to see what's going on and help train the system to think like a human.

Re: Differentiable Neural Computers

#72

Post by shostack »

visarga wrote: post #8
rkaplan wrote: post #5

This paper builds off of DeepMind's previous work on differentiable computation: Neural Turing Machines. That paper generated a lot of enthusiasm when it came out in 2014, but not many researchers use NTMs today. The feeling among researchers I've spoken to is not that NTMs aren't useful. DeepMind is simply operating on another level. Other researchers don't understand the intuitions behind the architecture well enou…

Deep mind is breaking new ground in number of directions. For example, "Decoupled Neural Interfaces using Synthetic Gradients" is simply amazing - they can make training a net async and run individual layers on separate machines by approximating the gradients with a local net. It's the kind of thing that sounds crazy on paper, but they proved it works. Another amazing thing they did was to generate audio by direct sy…

I'm not super knowledgeable about the space, but would the audio generation you mentioned be what is needed to let their Assistant communicate verbally in any language, any voice, add inflections, emotion, etc. without needing to pre-record all the chunks/combinations?

Re: Differentiable Neural Computers

#73

Post by shostack »

gugagore wrote: post #45
igravious wrote: post #32

Earlier quoted context omitted.

In this context what does differentiable mean?

I think the easiest way to see this is by an example of a non-differentable architecture. Let's suppose on the current training input, the network produces some output that is a little wrong. It produced this output by reading a value v at location x of memory. In other words, output = v = mem[x] It could be wrong because the value in memory should have been something else. In this case, you can propagate the gradien…

Pardon my ignorance as I'm not super knowledgeable on this, but is what you described around reading all the memory and taking the weighted sum of values similar in a sense to creating a checksum to compare something against?

Re: Differentiable Neural Computers

#74

Post by Senji »

dharma1 wrote: post #65
empath75 wrote: post #54

Earlier quoted context omitted.

Feeding mp3s to an image recognition neural net. And as soon as I typed that, I want to try it.

CNNs can actually be used for audio tasks too, on spectrograms

It's how some guys defeated the first iteration of recaptcha's audio mode. Then google replaced it with something very annoying to use even for humans.

Re: Differentiable Neural Computers

#75

Post by gugagore »

iandanforth wrote: post #61
gugagore wrote: post #45

Earlier quoted context omitted.

I think the easiest way to see this is by an example of a non-differentable architecture. Let's suppose on the current training input, the network produces some output that is a little wrong. It produced this output by reading a value v at location x of memory. In other words, output = v = mem[x] It could be wrong because the value in memory should have been something else. In this case, you can propagate the gradien…

Turing didn't specify how reads and writes happened on the tape. For the argument he was making it was clearer to assume there was no noise in the system. As for "digital" computers remember they are built out of noisy physical systems. Any bit in the CPU is actually a range of voltages that we squash into the abstract concept of binary.

I don't think that is really relevant to the discussion. Regardless of how a digital computer is physically implemented, we use it according to specification. We concretize the concept of binary by designing the machine to withstand noise. The thing what we get when we choose the digital abstraction is that this is actually realistic. Digital computers pretty much operate digitally. Corruption happens, but we consider that an error, and we try to design so that a programmer designing all but the most critical of applications, should assume that memory does not get corrupted

We don't squash the range of voltages. The digital component that interprets that voltage does the squashing. And we design it that way purposefully. https://en.wikipedia.org/wiki/Static_discipline

Turing specified that the reads and the writes are done by heads, which touch a single tape position. You can have multiple (finitely many) tapes and heads, without leaving the class of "Turing machine". But nothing like blending symbols from adjacent locations on the tape, or requiring non-local access to the tape.

Re: Differentiable Neural Computers

#76

Post by gugagore »

shostack wrote: post #73
gugagore wrote: post #45

Earlier quoted context omitted.

I think the easiest way to see this is by an example of a non-differentable architecture. Let's suppose on the current training input, the network produces some output that is a little wrong. It produced this output by reading a value v at location x of memory. In other words, output = v = mem[x] It could be wrong because the value in memory should have been something else. In this case, you can propagate the gradien…

Pardon my ignorance as I'm not super knowledgeable on this, but is what you described around reading all the memory and taking the weighted sum of values similar in a sense to creating a checksum to compare something against?

I suppose I can see the similarity, in that there's some accumulated value (the sum) from reading some segment of memory, but otherwise I don't think the comparison is helpful.

Re: Differentiable Neural Computers

#77

Post by ludoplex »

svantana wrote: post #21
modeless wrote: post #12

Earlier quoted context omitted.

The reason other researchers haven't jumped on NTMs may be that, unlike commonly-researched types of neural nets such as CNNs or RNNs, NTMs are not currently the best way to solve any real-world problem. The problems they have solved so far are relatively trivial, and they are very inefficient, inaccurate, and complex relative to traditional CS methods (e.g. Dijkstra's algorithm coded in C). That's not to say that NT…

They sure put a lot of focus on "toy" problems such as sorting and path planning in their papers - perhaps because they are easy to understand and show a major improvement over other ML approaches. IMHO they should focus more on "real" problems - e.g. in Table 1 of this paper it seems to be state of the art on the bAbl tasks, which is amazing.

Any chance you could link a pdf of the paper for us?
Post reply on HN