71–80 of 139 posts
Earlier quoted context omitted.
No, but it's also true that almost 40% of autists have intellectual disabilities: https://www.cdc.gov/mmwr/volumes/72/ss/ss7202a1.htm That said, the parent comment is just silly and wrong.
Autism, it turns out, is at least 4 different things: https://www.thetransmitter.org/spectrum/untangling-biologica...
After reading the article it seems to me that this is more like synaptic pruning where weak connections between neurons are eliminated in order to increase the efficiency of the neurons. Interesting to see that this also works for LLMs. https://en.wikipedia.org/wiki/Synaptic_pruning
The issue is that no one fully understands why synaptic pruning occurs in biology. Large language models have no direct connection to biological systems, and pruning in LLMs is no exception.
Earlier quoted context omitted.
> This would give LLM functioning autism Functioning autism hardly equals low intellect. Half the people of this forum (at least) are functioning autists.
I didn't say low intellect but , as also a functioning autist (as most of us are) i know myself that i am something wrong compare to other people who are quite different.
Earlier quoted context omitted.
The issue is that no one fully understands why synaptic pruning occurs in biology. Large language models have no direct connection to biological systems, and pruning in LLMs is no exception.
In ANNs pruning helps prevent over-fitting. With the discovery that transformers lack reasoning capabilities this research really comes at a great time. It's a miniscule chance, but we might see this improve performance over the long term and further research.
The only paper I have seen claiming this studied only lightweight open-source models (<27B, mostly 2B and 8B). The also included o1 and 4o for reference, which kind of broke their hypothesis, but they just left that part out of the conclusion. Not even kidding, their graphs show o1 and 4o having strong performance in their benchmarks, but the conclusion just focuses on 2B and 7B models like gemma and qwen.
It is significantly more complex than it appears at first sight.
Earlier quoted context omitted.
In ANNs pruning helps prevent over-fitting. With the discovery that transformers lack reasoning capabilities this research really comes at a great time. It's a miniscule chance, but we might see this improve performance over the long term and further research.
>With the discovery that transformers lack reasoning capabilities The only paper I have seen claiming this studied only lightweight open-source models (<27B, mostly 2B and 8B). The also included o1 and 4o for reference, which kind of broke their hypothesis, but they just left that part out of the conclusion. Not even kidding, their graphs show o1 and 4o having strong performance in their benchmarks, but the conclusio…
An 18% drop in accuracy (figure 8) is not insignificant. Even 4o suffered 10% loss (figure 6), and 4o isn't a small llm.
Competent performance should have near zero performance loss. The simplest benchmark merely changes things like "john had 4 apples" to "Mary had 4 oranges." Performance loss due to inconsequential tokens changing is the very definition of over-fitting.
Earlier quoted context omitted.
>With the discovery that transformers lack reasoning capabilities The only paper I have seen claiming this studied only lightweight open-source models (<27B, mostly 2B and 8B). The also included o1 and 4o for reference, which kind of broke their hypothesis, but they just left that part out of the conclusion. Not even kidding, their graphs show o1 and 4o having strong performance in their benchmarks, but the conclusio…
https://arxiv.org/abs/2410.05229 An 18% drop in accuracy (figure 8) is not insignificant. Even 4o suffered 10% loss (figure 6), and 4o isn't a small llm. Competent performance should have near zero performance loss. The simplest benchmark merely changes things like "john had 4 apples" to "Mary had 4 oranges." Performance loss due to inconsequential tokens changing is the very definition of over-fitting.
The authors did the equivalent of "Lets design a human intelligence benchmark, and use a bunch of 12 year olds as reference points"
I will eat my hat if the authors rescind the paper in a year or so if their benchmarks show no difference on SOTA models.
Earlier quoted context omitted.
In ANNs pruning helps prevent over-fitting. With the discovery that transformers lack reasoning capabilities this research really comes at a great time. It's a miniscule chance, but we might see this improve performance over the long term and further research.
>With the discovery that transformers lack reasoning capabilities The only paper I have seen claiming this studied only lightweight open-source models (<27B, mostly 2B and 8B). The also included o1 and 4o for reference, which kind of broke their hypothesis, but they just left that part out of the conclusion. Not even kidding, their graphs show o1 and 4o having strong performance in their benchmarks, but the conclusio…
Earlier quoted context omitted.
I think that's the intuition behind MoE (Mixture of Experts). Train separate subnets for different tasks, train a router that selects which subnets to activate at inference time. Mixtral is a current open model which I believe implements this.
No. MoE tends to change expert every other word. There’s a bit of pattern (like a lot of punctuation to one expert) but it’s not clear what. Nobody understands how or why the router chooses the expert. It’s so early.