Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Predictive Learning [pdf]

drive.google.com

Post reply on HN
Print view View original

31–40 of 41 posts

Re: Predictive Learning [pdf]

#31

Post by felippee »

raverbashing wrote: post #11

Very interesting set of slides It seems the current state of prediction is only slightly better than the state of image recognition pre multiple level NNs There might be still a theoretical jump that's needed

Yes, the field has emerged out of MNIST/ImageNet and that is what those algorithms are optimised for. For modelling actual dynamics different design is necessary. It happens that the design that makes sense also seems to agree very well with the observed biology of the cortex. You can find links to our Predictive Vision Model in this thread as well as a few additional thoughts here: http://blog.piekniewski.info/2016/11/30/learning-physics-is-...

Re: Predictive Learning [pdf]

#32

Post by TFortunato »

splike wrote: post #22
chessweb01 wrote: post #9

If page 33 depicts the working of the brain on a very high level, the world model (or simulator) residing inside the agent must contain a model/simulator of the agent itself. Could this give rise to self perception or consiousness?

I happened to be reading The Selfish Gene by Richard Dawkins and Gödel, Escher, Bach by Douglas Hofstadter at the same time, and both of them point at exactly this being the reason for consciousness. I was stunning at how both reached the same conclusion, that consciousness arrises from recursion of self perception, from very different points. Also, if anyone is watching Westworld ( spoilers ), it seems to come to th…

Re Westworld: The theory of consciousness explored in the show is explored in more detail in Jaynes' The Origin of Consciousness in the Breakdown of the Bicameral Mind, as alluded to both in the show and in the title of the final episode. I've just picked it up, and it's a pretty interesting read so far. I've also noticed that a lot of little details from the book used in the show, such as referring to memories as "reveries" at points, and talking of minds as "hosts" of consciousness. I may need to re-watch the show after finishing the book!

I found a pdf version of the book here if you are interested. http://selfdefinition.org/psychology/Julian-Jaynes-Origin-of...

Re: Predictive Learning [pdf]

#33

Post by eli_gottlieb »

pakl wrote: post #27

Lecun has identified a real problem for AI -- the need to understand the real world, the link between intelligence and prediction over time. But the tools he is using are not the right ones. Deep conv nets were not designed with prediction over time in mind. Here's one reason: Deep convolutional nets do not handle dynamical information from their lowest layers. By design, conv layers and pooling layers immediately be…

Yeah, not to be too dismissive or cliched, but these slides come off as if LeCun just now got around to reading some Andy Clark or reading neurosci/cogsci papers by Tenenbaum or Friston. The idea that the human brain has to work via active prediction rather than passive signal processing has been well-established in cogsci and neurosci for a while now.

The interesting question, then, is how we make machine-learning systems do prediction well. Probabilistic/generative models have been "wandering in the desert" for a while now because Monte Carlo methods are just so slow, especially for high-dimensional, hierarchical prediction problems like we want to solve in machine learning. On the upside, STAN now has automatic variational inference for continuous probability models, and work on things like the "concrete distribution" (https://arxiv.org/abs/1611.00712) can help us continuously approximate discrete probabilistic reasoning. Maybe as these techniques move into the mainstream in systems like Venture or Picture we can start to scale up predictive/generative/probabilistic modelling to match optimization-based connectionist methods?

Re: Predictive Learning [pdf]

#34

Post by felippee »

eli_gottlieb wrote: post #33
pakl wrote: post #27

Lecun has identified a real problem for AI -- the need to understand the real world, the link between intelligence and prediction over time. But the tools he is using are not the right ones. Deep conv nets were not designed with prediction over time in mind. Here's one reason: Deep convolutional nets do not handle dynamical information from their lowest layers. By design, conv layers and pooling layers immediately be…

Yeah, not to be too dismissive or cliched, but these slides come off as if LeCun just now got around to reading some Andy Clark or reading neurosci/cogsci papers by Tenenbaum or Friston. The idea that the human brain has to work via active prediction rather than passive signal processing has been well-established in cogsci and neurosci for a while now. The interesting question, then, is how we make machine-learning s…

The idea is not new indeed, Andy Clarks's review paper was very inspiring to us. But it has not been detailed to a point of implementation/scaling. PVM is an attempt to implement it in a "connectionist" way, but frankly all I need are associative memories, and how they are implemented I don't care. So a "probabilistic PVM" is totally feasible. In fact we discuss in the paper various possibilities in which a PVM like meta-architecture can be implemented.

Re: Predictive Learning [pdf]

#35

Post by lowglow »

hyperopia wrote: post #3
lowglow wrote: post #2

For anyone that doesn't know Yann LeCun, he's the head of AI over at Facebook, but surprisingly he's positively and consistently straightforward concerning the current hype driving AI and its technologies. He deserves respect because of this alone.

Your Asteria AI project looks pretty sweet.

Thanks, are you working in AI? You should join our gitter channel.

Re: Predictive Learning [pdf]

#36

Post by ceejay »

pakl wrote: post #27

Lecun has identified a real problem for AI -- the need to understand the real world, the link between intelligence and prediction over time. But the tools he is using are not the right ones. Deep conv nets were not designed with prediction over time in mind. Here's one reason: Deep convolutional nets do not handle dynamical information from their lowest layers. By design, conv layers and pooling layers immediately be…

Would it be feasible to replace the "common" component of a recurrent neural network with a convolutional neural network?

My lay person's impression is that at its most basic level a recurrent neural network is simply a "conveyor belt" of neural nets which are affected by external weights as well as by the weights from within the network. More precisely the "internal" weights coming from the layer of perceptrons operating at 1 level shallower than itself. So we're dealing in essence with 2 dimensions (shallower to deeper, and older to newer) instead of just one (shallower to deeper).

Re: Predictive Learning [pdf]

#37

Post by otoburb »

logancg wrote: post #14
chessweb01 wrote: post #9

If page 33 depicts the working of the brain on a very high level, the world model (or simulator) residing inside the agent must contain a model/simulator of the agent itself. Could this give rise to self perception or consiousness?

During the speech [1], Yann (surprisingly) didn't mention consciousness at all. The focus of this segment was the need to "imagine" the future. The premise is that "common sense" – Yann's big theme of the talk – is about "filling in the gaps" of incomplete information. We fill in the gaps by imagining the future. So consciousness was not raised at this point. But that doesn't mean that it couldn't be an emergent prop…

>>During the speech [1], Yann (surprisingly) didn't mention consciousness at all.

I thought this ommission was deliberate to avoid distracting philisophical ratholes that weren't core to his talk.

Re: Predictive Learning [pdf]

#38

Post by felippee »

ceejay wrote: post #36
pakl wrote: post #27

Lecun has identified a real problem for AI -- the need to understand the real world, the link between intelligence and prediction over time. But the tools he is using are not the right ones. Deep conv nets were not designed with prediction over time in mind. Here's one reason: Deep convolutional nets do not handle dynamical information from their lowest layers. By design, conv layers and pooling layers immediately be…

Would it be feasible to replace the "common" component of a recurrent neural network with a convolutional neural network? My lay person's impression is that at its most basic level a recurrent neural network is simply a "conveyor belt" of neural nets which are affected by external weights as well as by the weights from within the network. More precisely the "internal" weights coming from the layer of perceptrons oper…

Why would you want that? Conv net is not some magic. It's just a crude way to reduce dimensionality by loosing spatial location. For some things it works, for some it doesn't.

I think the shift in thinking should rather be: instead of trying to build the best possible associative memory to associate some A with some B, take the memory modules we have (perhaps not perfect) and try to build something bigger out of them. A dynamical model of the observed reality seems like a great thing to build out of such modules.

And this is what the PVM is. Currently made out of shallow, plain vanilla perceptrons, builds a structure which can be arbitrarily deep. Without any "magical" tricks such as dropout, relu, convolution, pooling etc.

Re: Predictive Learning [pdf]

#39

Post by jamesblonde »

escap wrote: post #5
hackinthebochs wrote: post #4

Earlier quoted context omitted.

Not to mention the father of CNNs.

it's a bit more complicated: https://en.wikipedia.org/wiki/Convolutional_neural_network#H... https://www.quora.com/Who-invented-convolution-neural-networ...

Who are you? Joergen Schmidthueber?

Re: Predictive Learning [pdf]

#40

Post by lowglow »

mac01021 wrote: post #24
lowglow wrote: post #2

For anyone that doesn't know Yann LeCun, he's the head of AI over at Facebook, but surprisingly he's positively and consistently straightforward concerning the current hype driving AI and its technologies. He deserves respect because of this alone.

"Straightforward" meaning what? Just bluntly dismissive of undue hype? What statements has he made to this effect?

Pretty much this in general. Check out any of his social profiles as he's pretty outspoken about his thoughts and opinions.
Post reply on HN