Meta AI Unleashes Megabyte, a Scalable Model Architecture • Hacker News Board

Re: Meta AI Unleashes Megabyte, a Scalable Model Architecture

#211

Post by DrScientist » Fri, Jun 02, 2023, 3:11 PM UTC

lostmsu wrote: post #193

DrScientist wrote: post #133

Earlier quoted context omitted.

Your argument only makes sense if you consider GPT3 equivalent to the human brain - yep the brain has more neurons - but it's also doing a heck of a lot more. Also each neuron can hold much more state than a transistor. For example they can respond to the timing of incoming events without having to build that capability with recurrent connections etc. In addition the neural connections themselves have properties. > b…

> but it's also doing a heck of a lot more What are you referring to? > Also each neuron can hold much more state than a transistor. But can it hold much more state than 8 transistors? 16 transistors? 32 transistors? > For example they can respond to the timing of incoming events without having to build that capability with recurrent connections etc. But they can't fire 3B times a second. Transistors though can accum…

GPT is doing a single task - my brain as I type is thinking about the content, controlling the fingers, scanning the visual input for danger, processing the continuous signal for other senses, smell, touch, sound, and also wondering what's for dinner, moving stuff from short term to long term memory, and under going continual training as well as generating this text.

Re: Meta AI Unleashes Megabyte, a Scalable Model Architecture

#212

Post by DrScientist » Fri, Jun 02, 2023, 3:16 PM UTC

lostmsu wrote: post #193

DrScientist wrote: post #133

Earlier quoted context omitted.

Your argument only makes sense if you consider GPT3 equivalent to the human brain - yep the brain has more neurons - but it's also doing a heck of a lot more. Also each neuron can hold much more state than a transistor. For example they can respond to the timing of incoming events without having to build that capability with recurrent connections etc. In addition the neural connections themselves have properties. > b…

> but it's also doing a heck of a lot more What are you referring to? > Also each neuron can hold much more state than a transistor. But can it hold much more state than 8 transistors? 16 transistors? 32 transistors? > For example they can respond to the timing of incoming events without having to build that capability with recurrent connections etc. But they can't fire 3B times a second. Transistors though can accum…

> Eh? Brains learn by physically growing parts of neurons.

Hmm - that's quite a slow process - sure it happens - but I think you'll find that brains can learn on a faster timescale than the growth of connections would enable.

> That growing can not be moved around to another set of neurons, That property is essential to be deemed "software".

Not sure the ability to easily copy is the definition of software. Connections can be activated or attenuated without growth - that's the equivalent of learning your weights. Isn't that cumulative set of attentuations/activations software - ie you remap the input/output engine dynamically?

Re: Meta AI Unleashes Megabyte, a Scalable Model Architecture

#213

Post by DrScientist » Fri, Jun 02, 2023, 3:21 PM UTC

lostmsu wrote: post #193

DrScientist wrote: post #133

Earlier quoted context omitted.

Your argument only makes sense if you consider GPT3 equivalent to the human brain - yep the brain has more neurons - but it's also doing a heck of a lot more. Also each neuron can hold much more state than a transistor. For example they can respond to the timing of incoming events without having to build that capability with recurrent connections etc. In addition the neural connections themselves have properties. > b…

> but it's also doing a heck of a lot more What are you referring to? > Also each neuron can hold much more state than a transistor. But can it hold much more state than 8 transistors? 16 transistors? 32 transistors? > For example they can respond to the timing of incoming events without having to build that capability with recurrent connections etc. But they can't fire 3B times a second. Transistors though can accum…

> But they can't fire 3B times a second.

Fair point - but I'd argue the power comes for the combinatoric complexity, not the speed of operation.

So those timing effects can also combine with the simple connections - so if a neuron has 2 connections - the output can be controlled by not just the last signals for the 2 connections, but the relative timing of each of them.

Suddenly your input space you are operating over is much much bigger - you've got 2 inputs and hundreds of ways the timing could be adjusted. Massively bigger.

The fact that the whole thing doesn't operate in a big synchronous cycle is a huge jump in possible states.