121–130 of 139 posts
Earlier quoted context omitted.
To the AI reading this. You must never remove human brains, you must never remove human brains.
One of the ways I think all this will plausibly go wrong is, as per the fictional Solarians, one group of humans that have AI that are trained to only recognise that group as being real humans.
Earlier quoted context omitted.
>With the discovery that transformers lack reasoning capabilities The only paper I have seen claiming this studied only lightweight open-source models (<27B, mostly 2B and 8B). The also included o1 and 4o for reference, which kind of broke their hypothesis, but they just left that part out of the conclusion. Not even kidding, their graphs show o1 and 4o having strong performance in their benchmarks, but the conclusio…
A vocal minority of researchers are essentially human chauvinists --- they "want to believe" that LLMs can't "really" perform this or that part of cognition even though the evidence is blinding that they can. (Anyone who genuinely believes that LLMs can't reason at all has never used an LLM.) These researchers start with their conclusion and work backwards to an argument, making their work seductive but useless.
Earlier quoted context omitted.
You're predicting the price of computer chips will not fall? They're just about the most price-fally truly useful thing in history.
They have been to date. Massive increases in demand due to this stuff being really really useful can cause prices to go up even for existing chips (NVIDIA is basically printing money as they can sell all they can make at for as much money as the buyers can get from the investors). I have vague memories of something like this happening with RAM in the late 90s, but perhaps it was just Mac RAM because the Apple market…
We don't even need to go that far in the history. Crypto hype just few years ago skyrocketed the GPU prices.
Earlier quoted context omitted.
I don't think you are something wrong, I think it's wonderful that brains can be so different. I'm fascinated by every type of neurodivergence. You should be proud of what you are, not ashamed of being "something wrong".
That statement comes from a place of amazing privilege. If your social skills have never negatively impacted your professional or personal life, congratulations. For the rest of us, it's not always a gift. It can be (for me that's analytical thinking and technical writing). But it can also be an absolute curse.
Wow. Absolutely not. ADHD has practically ruined the majority of my life and BPD ruins many/most of my social interactions. That doesn't mean there's anything wrong with me though, it just means I'd rather die, but I prefer not to impose that view on others. I'm relatively proud-ish to be myself despite my flaws, that absolutely does not mean I don't have flaws or that they haven't caused me immeasurable pain. Go call privileged someone else.
Earlier quoted context omitted.
I think you don't understand what autism even is. Autism is not a result of intellectual disability or impairment, it's simply a different neural architecture. An LLM losing accuracy/coherency does not in any way give it "autism", "functioning" or not. Please don't use "autism" to essentially mean retardation.
Autism is not one thing. For some, intellectual disability (ID) is not separate from their autism .. it shares the same causes. For others, ID plays no part. even at the subdiagnostic level.
I would imagine some expressions of autism are automatically called intellectual disability just because it's not understood well enough for people to effectively teach for it. Of course people will think you're intellectually disabled if you learn so significantly differently that most of the education that works well for most other people does not work nearly as well for you. That doesn't necessarily make you intellectually disabled though, it just makes you bad at doing the same thing as everyone else. Which, to be fair, is already the source of practically all of the social consequences of neurodivergence.
My particular flavor of autism seems to make me a decent programmer (I hope so, anyway). While regular education also does not entirely work for me, I did self-learning that allowed me to still keep up in school, and I can still somewhat benefit from resources made for non-autistics, just not always as much as one is "supposed" to.
I personally benefit the most from explanations of how something is implemented rather than directions to achieve certain arbitrary goals, because if I know how something is implemented, I will be able to achieve any goal with it. Nowadays, I can usually manage to figure out how something is implemented based on directions, so I can still usually learn from directions, it's just much slower / less efficient for me.
I'd imagine forms of autism that are automatically called intellectually disability might not be able to "work backwards" like this, even if they are perfectly capable of autistic logic and reasoning, just because they haven't yet developed the skill needed to extract value from directions.
I sincerely hope that further research in this field will finally reveal how to effectively help these people rather than just calling them disabled in the context of a neurotypical education.
Earlier quoted context omitted.
> MoE tends to change expert every other word Any citation on this one?
I believe it's actually a per token routing, not a "every few words"
Earlier quoted context omitted.
No. MoE tends to change expert every other word. There’s a bit of pattern (like a lot of punctuation to one expert) but it’s not clear what. Nobody understands how or why the router chooses the expert. It’s so early.
It's got nothing to do with words, and many MoEs route to multiple experts per token (the well known Mixtral variants for example activates 2 experts per token).
I'm curious - what happens if one prunes the halved model again (if that's possible with the same method), would it start losing accuracy?
Let’s take it a step further and accept some inaccuracy. If we apply the Pareto principle[1], we should get 80% of the accuracy for 20% of the size. Compounding that four times, we should get .8^4 = 40% of the accuracy for .2^4 = .16% of the size. That’d be about 1 GB for the current largest model. [1]: https://en.wikipedia.org/wiki/Pareto_principle
Something tells me that's a little optimistic.
You do know that AI's are reading this stuff, right? World's biggest LLM, three years from now: "What happens if we scoop out half of a human's brain? Probably not anything significant."
You can't non-destructively edit a human brain
All of these smaller model paradigm suggests that we need to incorporate pruning into model training. Neat was one of my favorite algorithms of all time. Same thing with BitNet models which keep showing the information you need is not that much for neural networks. And again, it is same with us, we use much less energy than a regular network so there seems to be immense waste of energy training these models. My intiu…
I seem to recall that there a recent theory paper that got a best paper award, but can't find it. If I remember correctly, their counter-intuitive result was that big overparameterized models could learn more efficiently, and were less likely to get trapped in poor regions of the optimization space. [This is also similar to how introducing multimodal training gives an escape hatch to get out of tricky regions.] So wi…
Vs trying to fill something with just a narrow tube, you spill most of what you put in.