131–140 of 293 posts
> so the model figures out during training what each token should look for and what it should offer But how does it learn this token-relationship? All it has is many text samples, but still, nowhere it says how the tokens relate to each other, so where does this information come from?
Earlier quoted context omitted.
normal people talk and write with some notion of meter, the cadence of communicating where pauses are inserted at places that naturally suit the speaker (and listener) to pause for thought. LLM's don't really do that, they just write a bunch of sentences. > Researchers have found that some neurons inside the FFN are strongly associated with specific concepts or facts. One neuron might activate strongly on Eiffel-Towe…
I don’t disagree with your conclusion that this is likely ai rewritten, but I do find it strange that you say “normal people don’t write like this” when it is mimicking how people write, and using patterns I have seen people write. I think models are at the point where style is not really reliable as an indicator anymore.
Earlier quoted context omitted.
normal people talk and write with some notion of meter, the cadence of communicating where pauses are inserted at places that naturally suit the speaker (and listener) to pause for thought. LLM's don't really do that, they just write a bunch of sentences. > Researchers have found that some neurons inside the FFN are strongly associated with specific concepts or facts. One neuron might activate strongly on Eiffel-Towe…
I don’t disagree with your conclusion that this is likely ai rewritten, but I do find it strange that you say “normal people don’t write like this” when it is mimicking how people write, and using patterns I have seen people write. I think models are at the point where style is not really reliable as an indicator anymore.
Earlier quoted context omitted.
In the 90s you didn't have norm layers, residuals, attention, and some more. So you're missing a lot of the building blocks that make LLMs. It's not a matter of just having the compute.
I think the attention mechanism is so simple but so revolutionary that people forget it. Like the best leaps in thinking, once it is made, is is immediately obvious and intuitive.
Residual connections are so simple, so obvious and so vital. Yet nobody came up with them until 2015?
Earlier quoted context omitted.
We don’t treat children like they’re stupid, we treat children like they’re children. A stupid adult is treated very differently than any child. Adults are expected to have their world models approximately correct in terms of physical environment so they won’t accidentally kill themselves by falling off a cliff; then there are the social norms which adults are expected to conform to so everyone is kinda predictable t…
You may have been raised properly since you don’t get what I mean. I really envy kids with “Chinese parents” that had them learn math early on and not some bullshit like that if you put your tooth under your pillow, then a tooth fairy will come.
Earlier quoted context omitted.
Because god forbid that childhood, the one time in your life when you don't have any responsibilities, should be fun.
Waste 22 years of life without learning anything and then slave away at a 9-5 job you hate. Brilliant strategy. At least you had “fun”. Then blame billionaires or something.
Earlier quoted context omitted.
We do know how they work. They predict the next statistically most likely token. The "bitter lesson" is that fake-it-till-you-make-it is a valid way of doing knowledge work. (Or not make it, then people will just claim you're holding the LLM wrong and it's not the AI's fault.)
> statistically most likely token. Statistically most likely in what context, given which preconditions? Because each prompt sequence is unique so the probability of any token following it is unknown.
Earlier quoted context omitted.
There is a different way to look at this: that is, actually the Transformer is a minimal complication of what the based model is: in theory the neural network could be just a huge FFN, which is anyway the part of the Transformer that does the heavy lifting. But this would be impossibile to train both numerically and computationally, so the Transformer encodes enough priors for it to work: the causal attention, and th…
Isn't that over-simplifying it a bit too much? You can go another step - a FFN can be simulated on a Turing machine, thus it just exemplifies the incredible semantical power of the Turing machine model of computation. (in fact you don't even need a Turing machine, since there is no looping in one forward pass). In theory you can run a huge FFN on the tiniest Turing machine, in practice it's much better to run a Trans…
Earlier quoted context omitted.
It's still predicting the next word. Somewhere in the gigantic dataset that the LLM was trained on, there is a phrase that says "gradient border" being in the vicinity of a CSS code that render the stuff. Therefore when you run it on an inference loop there's a good chance it output that CSS code when you tell it to render a "gradient border" Multi-modal models that can understand visual input do exists, but no such…
What about things it wasn’t trained on? For instance I’ve written a few custom languages to learn how to write a VM and the lexer/parser/compiler/etc. that it had never seen before and then just gave it the syntax which is different than what it had ever seen before. Simply due to the fact I made it and it had never been trained on it. After giving it my documentation, it was able to write the language just like a la…
The exact syntax does not matter, only the grammar. If you give it the grammar, and then the keywords, it can find something that has similar grammar and then use your keywords.
Earlier quoted context omitted.
> statistically most likely token. Statistically most likely in what context, given which preconditions? Because each prompt sequence is unique so the probability of any token following it is unknown.
It’s not unknown because that’s what the model computes. It’s matrix multiplication just like shaders.