41–50 of 100 posts
Training data has 0.339T tokens, less than the number of training parameters. A model like that could store all of the training text with 100B+ parameters left for computation.
Will anyone outside of Nvidia be able to access it? GPT-3 at least has an API.
Earlier quoted context omitted.
Except that every synapse is not a dumb weight but a highly complex system connected to an even more complex system (aka neuron) which might each be a (super)computer on its own. Given how extremely bad we are at computing, there is hope (for ai) that the neurons or their circuits are not _that_ powerful after all.
Last I heard (and I believe this could be wrong) my professor said that we basically understand how a single neuron works. That like basically if we do X input we get Y output, up to some accuracy. He used this to discuss the idea behind neural networks -- that each neuron is simple enough to model, all we need to worry about is the weights and the dynamics of the network as a whole. How much of a simplification is t…
Such research is the area of computational neuroscience - one thing that such people do is try to model parts of the brain (or just a single neuron) with computers.
A Neuron (=nerve cell in the brain) is a very complex beast. In rough terms they work like this: They collect signals (electrical impulses) via their small appendages called dendrites. when the sum of the signals reaches a certain threshold a large electrical impulse is generated at the cell body that will travel trough its "output" appendage (called axon) that connected to another neuron's cell body or to its dendrite.
Neurons display a dazzling variety in all these parameters:
- In morphology, e.g. they can look like a pine tree http://www.scholarpedia.org/article/Pyramidal_neuron (I really recommend scholarpedia, also this article has a nice animation on how electrical impulses propagate) or like a sea urchin.
- it really matters where the cell gets its impulse from: A neuron stimulated near its cell body will be much more sensitive to the input than being stimulated far away.
- Their response characteristics are wildly varied too. Some give off one large impulse, some a quick burst of impulses. Some are preventing others from giving out impulses from stimulation (inhibitor neurons)
- This whole mess can be modulated with chemical compounds that are released by the body -- some make some neurons more sensitive, some less.
- Also we still discover every year some new mechanism that modulates how they function.
The issue is that this results in such a complex system that a modern PC cant even simulate 1 detailed neuron model realtime (these tools are open source, try them out! for example https://neuron.yale.edu/ ). Now we know that we're simulating things that likely do not matter (e.g. we don't need a neuron model that consist of 10.000+ segments), but we do not know which parts we need to remove to have a faithful simulation. Also we might simply simulate some parts wrong because our knowledge of the subject is not enough.
But on the upside we've reached some great things already, for example we know how our brain calculates from our head and eye position the orientation of the things we're looking at
What's really interesting is that these models are using some non-trivial portion of all easily accessible human writing -- yet humans learn language really well with significantly less input data. What's missing in the field to replicate human performance in learning?
Training data has 0.339T tokens, less than the number of training parameters. A model like that could store all of the training text with 100B+ parameters left for computation.
Earlier quoted context omitted.
Last I heard (and I believe this could be wrong) my professor said that we basically understand how a single neuron works. That like basically if we do X input we get Y output, up to some accuracy. He used this to discuss the idea behind neural networks -- that each neuron is simple enough to model, all we need to worry about is the weights and the dynamics of the network as a whole. How much of a simplification is t…
> All we need to worry about is the weights and the dynamics of the network as a whole. How much of a simplification is that? A lot. Parallel optimization is an art form. These models are trained on static datasets, they can't intervene in the environment to infer causal relations, so they need legs and hands.