Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

DeepMind’s new AI with a memory outperforms algorithms 25 times its size

singularityhub.com

Post reply on HN
Print view View original

121–130 of 141 posts

Re: DeepMind’s new AI with a memory outperforms algorithms 25 times its size

#121

Post by katmannthree »

SubiculumCode wrote: post #114
katmannthree wrote: post #50

Earlier quoted context omitted.

I believe we've been able to model the processes behind neural firings for quite some time now, the issue is more mapping between that micro-scale understanding of how each individual piece in a 10^10 piece puzzle with 10^15 edges affects the macro-scale emergent properties of said puzzle. It's less a lack of understanding of the processes and more a complexity so great that it exceeds the grasp of our best modeling…

Our biophysicalmodesl of neurons are actually woefully simplistic. https://journals.physiology.org/doi/full/10.1152/jn.00360.20...

Yes. The simple models are simple, the fast and large and useful models are simple. We have a lot of in vitro observations which can contribute to smaller, slower and more accurate models as your linked article mentions, observations which generally are ignored as being nth order effects when people are trying to make models which have a low enough computational complexity as to be useful.

What is not missing is metaphysical quantum woo[0]. We can, have, and do observe how neurons and synapses function; the issue is that the computational complexity with current approaches is great enough to make complete neurological modeling of a microscopic worm with under a thousand total cells difficult.

[0] and even if you want to take those effects into account in your electrochemical models, all they do is turn them into stochastic models. there is exactly zero evidence of that randomness being of any real value to the thing we care about, which is the emergent macro scale properties of these systems.

Re: DeepMind’s new AI with a memory outperforms algorithms 25 times its size

#122

Post by ummonk »

blurbleblurble wrote: post #117
saltyfamiliar wrote: post #19

Earlier quoted context omitted.

Does anyone know if we understand enough about natural, generally intelligent brains to dismiss the idea that they are using quantum phenomenon for computation? Is it unlikely for any reason?

There's actually a whole field of study that uses tools from quantum information theory to successfully model cognitive and social situations. This doesn't require any assumptions of "quantum physicality". It just makes sense to model probabilistic, reflexive, slippery-to-observe systems with tools that were designed for this. Quantum theory has really great tools and they work well. I realize this isn't quite the an…

Surely they'd just use tools from regular information theory and Bayesian inference? Quantum theory is just one particular application of information theory, where uncertainty is directly controlled by physical processes rather than simply due to imperfect knowledge.

Re: DeepMind’s new AI with a memory outperforms algorithms 25 times its size

#123

Post by ummonk »

neom wrote: post #32
torbTurret wrote: post #13

Earlier quoted context omitted.

What’s your argument?

I hope I didn't come off like I think know anything about this field, because I don't. A friend who use to work for openai (Jack Clark) and I spend some time once discussing over beers some stuff around general purpose AI, and I proposed that I believe quantum is a dark horse on the road to general purpose AI, and he disagreed.

Yeah he's right, you're wrong.

Re: DeepMind’s new AI with a memory outperforms algorithms 25 times its size

#124

Post by tasty_freeze »

kkjjkgjjgg wrote: post #53

Sounds as if they stored all the correct answers in a database and call it "better". How do they even evaluate these models? Like they already have a billion preprepared correct answers in the database. How do they come up with new questions for the evaluation?

> all the correct answers

That is clearly not possible, so it can't be what they are doing.

Rather than diffusely encoding that knowledge in a massive number of self-organized layers of weights, it is explicitly encoded. The remaining network can "focus" on mapping input to retrieve the relevant information stored in that database, and extracting/interpolating/extrapolating that information based on the current context to generate useful output.

Re: DeepMind’s new AI with a memory outperforms algorithms 25 times its size

#125

Post by machiaweliczny »

saltyfamiliar wrote: post #19
neom wrote: post #12

Earlier quoted context omitted.

I got into a very long debate with an openai person 4/5 years ago about this + adversarial learning + access to a quantum computer (think just straight up world class abacus) was close to the primitives required for more generalized AI. They didn't agree with me, but that's ok! :)

Does anyone know if we understand enough about natural, generally intelligent brains to dismiss the idea that they are using quantum phenomenon for computation? Is it unlikely for any reason?

We will only know if we can't built it.

My gut feeling is that even if we use some probabilistic quantum compute it should be transferable to normal compute. Also animals vs human don't have enough difference to assume we are special.

Re: DeepMind’s new AI with a memory outperforms algorithms 25 times its size

#126

Post by kkjjkgjjgg »

blovescoffee wrote: post #111
kkjjkgjjgg wrote: post #53

Sounds as if they stored all the correct answers in a database and call it "better". How do they even evaluate these models? Like they already have a billion preprepared correct answers in the database. How do they come up with new questions for the evaluation?

Instead of storing the correct answers in an encoded/embedded form in the weights of the neural net (certain neurons very loosely corresponding to certain "answers") the correct answers are stored elsewhere. That way we can scale down the model to the necessary "thinking" parts and we don't need to use excess neurons for the "memory" part. Kind of handwavey but hopefully that explains the general idea.

You mean otherwise the whole words would be encoded in the net, and now you only need to encode the index in the database?

Re: DeepMind’s new AI with a memory outperforms algorithms 25 times its size

#127

Post by [deleted] »

whazor wrote: post #15

Very interesting. GPT-J is an opensource free alternative to GPT-3 and requires at least 12.1GB memory to run the model (which is reduced from original 48GB ram). But if the model stores some kind of index and does internet searches (or hard drive) instead, then it could scale much further as there is a limit on how much memory you can use in production.

[deleted]

Re: DeepMind’s new AI with a memory outperforms algorithms 25 times its size

#128

Post by blurbleblurble »

ummonk wrote: post #122
blurbleblurble wrote: post #117

Earlier quoted context omitted.

There's actually a whole field of study that uses tools from quantum information theory to successfully model cognitive and social situations. This doesn't require any assumptions of "quantum physicality". It just makes sense to model probabilistic, reflexive, slippery-to-observe systems with tools that were designed for this. Quantum theory has really great tools and they work well. I realize this isn't quite the an…

Surely they'd just use tools from regular information theory and Bayesian inference? Quantum theory is just one particular application of information theory, where uncertainty is directly controlled by physical processes rather than simply due to imperfect knowledge.

Quantum information comes from Von Neumann entropy, while classical information comes from Shannon entropy. Regardless of where the "source" of information/uncertainty comes from or how "physical" it is, they're related but distinctly different tools which can each be applied to both "physical" or "non-physical" systems at any scale.

"Quantum" is turning out to have very little to do with how tiny or physical stuff is, and much more to do with assumptions about how observations emerge from interaction.

> rather than simply due to imperfect knowledge

That's the key right there... Quantum probability and quantum models don't rely on the existence of "perfect knowledge" or "underlying" objective states. States themselves are intrinsically probabilistic and contextually embedded. Measurements/observations cannot be cleanly decoupled from the states being measured, and are modelled as projections from a high dimensional space of possibilities onto some lower dimensional subspace.

This kinda thing works really well for social/cognitive systems which are incredibly sensitive to measurement process. For example, when conducting polls or surveys, the ordering of the questions is well known to impact the outcome. It turns out that this can be very well modelled using tools from quantum theory, and it has been.

Check out this book and all the books/papers citing it for a window into this fascinating world

https://link.springer.com/book/10.1007/978-3-642-05101-2

Re: DeepMind’s new AI with a memory outperforms algorithms 25 times its size

#129

Post by visarga »

HPsquared wrote: post #104
visarga wrote: post #91

Earlier quoted context omitted.

Not necessary. Quantum AI is basically explaining something mysterious by something else mysterious, even when there is no indication they are related.

Aren't a lot of AI breakthroughs like this though? "It works, but we don't know why"

Good point. But at least we know the ML algorithms are related to ML performance, we don't know that about QM.

Re: DeepMind’s new AI with a memory outperforms algorithms 25 times its size

#130

Post by torbTurret »

netizen-936824 wrote: post #85
neom wrote: post #32

Earlier quoted context omitted.

I hope I didn't come off like I think know anything about this field, because I don't. A friend who use to work for openai (Jack Clark) and I spend some time once discussing over beers some stuff around general purpose AI, and I proposed that I believe quantum is a dark horse on the road to general purpose AI, and he disagreed.

I think the person you're replying to is asking for your specific argument. As in what exactly do you think is the advantage gained by quantum computing and why is it important.

Yeah I have no horse in this race, just curious of your viewpoint as someone who works in the field!
Post reply on HN