How deep is the brain? The shallow brain hypothesis

Post by jacobsimon » Tue, Oct 31, 2023, 3:02 AM UTC

titzer wrote: post #169

audunw wrote: post #115

I seem to remember research stating that an individual neuron has very complex behaviour that requires several ML “neurons” / nodes to simulate. So if you do a comparison, perhaps the brain is deeper than you’d think by just looking at the graph of neurons and their synapses. Could we construct a neutral net from nodes with more complex behaviour? Probably, but in computing we’ve generally found that it’s best to bui…

Yeah, biological brains could be remarkably more powerful than digital neural networks if the have primitive functions that we haven't accounted for. For example, some networks seem to encode information in the firing rate , rather than just the presence of a signal. If neurons could, e.g. do frequency-based calculations (and not just threshold-based, like spiking neural nets), they could be orders of magnitude more…

Post by edmundsauto » Fri, Nov 03, 2023, 4:51 PM UTC

4death4 wrote: post #34

edmundsauto wrote: post #22

Earlier quoted context omitted.

I don’t think this is an apt comparison, but I do think the amount of energy it takes to grow a human into brain maturity in adulthood is an interesting one. Brains + bodies over a 20 year development cycle is still probably much less than training even a low quality Llm.

Let’s say a human needs an average of 2000 calories a day. A calorie is roughly equivalent to 1 Watt hour, so over 20 years, it takes about 15 MWh to sustain a human. Let’s say a single A100 has a peak power draw of 250W, and you need 100 to train an LLM. So each hour of training consumes 25,000 Wh of energy. 15 MWh / 25,000 W = 600 hours, or 25 days, which is probably pretty close to the true training time. So the n…