Averaging is a convenient fiction of neuroscience • Hacker News Board

Re: Averaging is a convenient fiction of neuroscience

#81

Post by _yb2s » Tue, Sep 24, 2024, 4:59 PM UTC

Jweb_Guru wrote: post #53

llm_trw wrote: post #38

Earlier quoted context omitted.

>Sounds like we mostly agree, save some pedantry. If by pedantry you mean that I'm not ignoring the cause of 70% of the improvement in life expectancy in the last 200 years then sure.

Indeed, this conversation is a good illustration of the damage that Bayesian statistics have done to the "educated" populace. Not that they're inherently bad--it's just a different statistical approach, and it's generally good not to assume a universal background, that everything is normal, etc.--but by telling people it's fine to question statistical conclusions because the distribution might be different, it libera…

I realize this is a bit of a "no true Scotsman" but what you are talking about is a gross misuse of Bayesianism- where your own biases are incorrectly treated as extremely strong evidence.

I am partial to using unform priors over all possibilities, and then just adding in the actual evidence for which you can actually quantify its quality/strength. Your "prior" for a new situation is constructed by applying the data you already had previously to a uniform prior- not by fabricating it from whole cloth via your biases. In practice this may be impossible for humans to do in their heads, but computers certainly can!

Re: Averaging is a convenient fiction of neuroscience

#82

Post by _yb2s » Tue, Sep 24, 2024, 5:04 PM UTC

domofutu wrote: post #64

_yb2s wrote: post #7

I've become increasingly convinced that the idea of averaging is one of the biggest obstacles to understanding things... it contains the insidious trap of feeling/sounding "rigorous" and "quantitative" while making huge assumptions that are extremely inappropriate for most real world situations. Once I started noticing this I can't stop seeing this almost everywhere- almost every news article, scientific paper, etc.…

Averages can definitely oversimplify things, especially in neuroscience where outliers often tell the real story. Taleb touches on this in Antifragile—focusing too much on the average can make us miss what’s happening at the edges, where the most interesting things are. Instead of leaning on averages, we might get more insight by paying attention to the extremes, where the real nuances are hiding.

Yes, I like Taleb's suggestion of considering a "fat tailed" Cauchy distribution- as to not assume extreme outliers can't exist

Re: Averaging is a convenient fiction of neuroscience

#83

Post by markhahn » Tue, Sep 24, 2024, 5:36 PM UTC

yberreby wrote: post #74

robertclaus wrote: post #17

As a computer scientist, I was blown away the first time my friend explained to me that his research focused on the timing of neuron spikes, not their magnitude. After talking about it for a while I realized that machine learning neural networks are much closer to simple early models of how neuron's work (averages and all), not how neuron's actually signal. Makes sense when you consider how the latest LLM models have…

> the latest LLM models have almost as many parameters as we have neurons I often see this take, but the apt comparison is between parameter and synapse count, not neuron count. You should be counting hidden units rather than weights if you want to compare to neuron counts.

I think you should compare synapse and weight counts if you want a measure of the network's state/capacity. If you want something closer to its compute power, compare neurons and hidden units.

Re: Averaging is a convenient fiction of neuroscience

#84

Post by smokel » Tue, Sep 24, 2024, 5:41 PM UTC

ithkuil wrote: post #63

robertclaus wrote: post #17

As a computer scientist, I was blown away the first time my friend explained to me that his research focused on the timing of neuron spikes, not their magnitude. After talking about it for a while I realized that machine learning neural networks are much closer to simple early models of how neuron's work (averages and all), not how neuron's actually signal. Makes sense when you consider how the latest LLM models have…

An elephant brain has 3 times as many neurons as a human. They are pretty smart animals but so are dogs who have way less neurons. The point here being that the number of neurons is just one of the many factors that determines intelligence (general or not)

Even among humans with roughly similar neuron counts, there are notable examples of individuals displaying extreme stupidity.

Re: Averaging is a convenient fiction of neuroscience

#85

Post by SkyBelow » Tue, Sep 24, 2024, 5:44 PM UTC

llm_trw wrote: post #19

_yb2s wrote: post #7

I've become increasingly convinced that the idea of averaging is one of the biggest obstacles to understanding things... it contains the insidious trap of feeling/sounding "rigorous" and "quantitative" while making huge assumptions that are extremely inappropriate for most real world situations. Once I started noticing this I can't stop seeing this almost everywhere- almost every news article, scientific paper, etc.…

>In reality the modal life expectancy of adults (most common age of death other than 0) has been pretty stable in the 70s-80s range for most of human history- the low average was almost entirely due to infant mortality. This is even wronger than what you critique. For every period in history that we have good data for people had a half-life - a period in which you'd expect half of all people to die: https://batinthea…

Is there an issue with how the data is grouped? At 19.9, you have a 50% chance of living to 59.9. But less than a season later at 20.1, you now only have a 50% chance o living to 40.1. How can the former be right if the latter is right?

The growth in life expectancy from surviving early childhood makes sense, but the decline in life expectancy for crossing 20 feels too sharp.

Re: Averaging is a convenient fiction of neuroscience

#86

Post by Nahtnah » Tue, Sep 24, 2024, 5:45 PM UTC

lamename wrote: post #32

robertclaus wrote: post #17

As a computer scientist, I was blown away the first time my friend explained to me that his research focused on the timing of neuron spikes, not their magnitude. After talking about it for a while I realized that machine learning neural networks are much closer to simple early models of how neuron's work (averages and all), not how neuron's actually signal. Makes sense when you consider how the latest LLM models have…

Yes and no. An alternate perspective is that the output of each neuron in an artificial neural net is analogous to an F-I curve in a real neuron (spike frequency-input DC current curve). In this way, different neurons have different slopes and intercepts in their FI curves, just as a network of ANN neurons effectively have their activation functions tweaked after applying weights. I usually only say this to other neu…

Eh, the analogy is farrr from perfect. You're basically assuming you can reduce neurons to LTI systems. Which you obviously very much cannot.

Re: Averaging is a convenient fiction of neuroscience

#87

Post by authorfly » Tue, Sep 24, 2024, 6:04 PM UTC

llm_trw wrote: post #60

_yb2s wrote: post #52

Earlier quoted context omitted.

It’s the right thing to use in this specific context not because it is explaining the full picture better than the average, but because it distorts the picture in a different way: the modal age being high is incompatible with the incorrect assumption people are making when seeing a low average life span- that people in ancient times basically never made it to an old age in their 70s or older. People then jump from th…

>because it distorts the picture in a different way: the modal age being high is incompatible with the incorrect assumption people are making when seeing a low average life span- that people in ancient times basically never made it to an old age in their 70s or older. I literally proved by demonstration why having a distribution with an absolute mode in the extreme most value means that 97% of people still died young…

The paper he linked though isn't a distribution where the age 72 is a special modal number. The paper suggests 68-78 is the adaptive lifespan for some small communities.

Re: Averaging is a convenient fiction of neuroscience

#88

Post by _yb2s » Tue, Sep 24, 2024, 6:05 PM UTC

Mistletoe wrote: post #35

_yb2s wrote: post #7

I've become increasingly convinced that the idea of averaging is one of the biggest obstacles to understanding things... it contains the insidious trap of feeling/sounding "rigorous" and "quantitative" while making huge assumptions that are extremely inappropriate for most real world situations. Once I started noticing this I can't stop seeing this almost everywhere- almost every news article, scientific paper, etc.…

I was at the Denver museum’s mummy exhibit and disturbed to see that they said the lady died at 30, a “normal age for death in those times”. You would think a museum should know better. https://aeon.co/ideas/think-everyone-died-young-in-ancient-s...

Great article that pretty well explains the situation with modern vs ancient life expectancy.

Re: Averaging is a convenient fiction of neuroscience

#89

Post by stonethrowaway » Tue, Sep 24, 2024, 6:07 PM UTC

hinkley wrote: post #15

There's an old case study from aerospace that shows up sometimes in UX discussions, where the US military tried to design an airplane that fit the 'average' pilot and found that they made a plane that was not comfortable for any pilots. They had to go back in and add margins to a bunch of things, so they were adjustable within some number of std deviations of 'average'.

They used those original average measurements to design seats for passengers instead.

Re: Averaging is a convenient fiction of neuroscience

#90

Post by giardini » Tue, Sep 24, 2024, 6:42 PM UTC

Nahtnah wrote: post #86

lamename wrote: post #32

Earlier quoted context omitted.

Yes and no. An alternate perspective is that the output of each neuron in an artificial neural net is analogous to an F-I curve in a real neuron (spike frequency-input DC current curve). In this way, different neurons have different slopes and intercepts in their FI curves, just as a network of ANN neurons effectively have their activation functions tweaked after applying weights. I usually only say this to other neu…

Eh, the analogy is farrr from perfect. You're basically assuming you can reduce neurons to LTI systems. Which you obviously very much cannot.

LTI=Linear Time Invariant

https://en.wikipedia.org/wiki/Linear_time-invariant_system