11–20 of 27 posts
> GPT-4 boasts 100 trillion parameters Source? I’ve never heard this. I’ve heard 1 trillion and that it’s a 8 x 175B ensemble.
I don't think it's particularly helpful to dive into transformer models, positional encodings and self-attention at the very start of an introduction to LLMs. Understanding how those works does little to help explain what LLMs can do and how you can use them. I tend to stick with the higher level explanation that they can predict the next word (or next sentence) based on their training text, and then emphasize that w…
I don't think it's particularly helpful to dive into transformer models, positional encodings and self-attention at the very start of an introduction to LLMs. Understanding how those works does little to help explain what LLMs can do and how you can use them. I tend to stick with the higher level explanation that they can predict the next word (or next sentence) based on their training text, and then emphasize that w…
any recommendations for explainers that operate on this level of abstraction?
Earlier quoted context omitted.
I'm not sure it's even helpful to talk about transformer at all[0]. Modern LLMs are different than the pure transformer architecture in many detailed aspects. The elephant in the room being that they're not even full transformer—only the decoder part. [0]: One reason: Never once did I need to know the transformer architecture in order to be able to use these models (prompt engineering, chaining, working with local mo…
>I think we need something that at least captures the general mechanism by which LLMs predict the next token. This bit varies a lot since the capabilities involved in prediction depend on the data. If the text is a math book and the prompt is "... three plus five apples is a total of " the crucial capability is arithmetic (plus of course NL capabilities). On the other hand, if you're completing a post from /r/relatio…
A language model trained on enough text will be able to complete this just based on having encountered the pattern "three plus five SOMETHING is a total of..." enough times in its training data.
This becomes even more apparent when you work with smaller models - the 7B etc models which can run on a laptop. They can often solve small arithmetic problems like this while having no chance at all of working with larger numbers that they haven't ever encountered in their training data.
I really like using those smaller models as tools to better understand how this technology works.
I don't think it's particularly helpful to dive into transformer models, positional encodings and self-attention at the very start of an introduction to LLMs. Understanding how those works does little to help explain what LLMs can do and how you can use them. I tend to stick with the higher level explanation that they can predict the next word (or next sentence) based on their training text, and then emphasize that w…
I wanted to talk about what powers LLMs, which I believe is important. The answer to that is transformers. While I may not have delved deeper into how a transformer actually works, I tried to explain the concepts in the simplest way possible.
Earlier quoted context omitted.
Almost like an AI written article
We have many existing sources on LLMs. I referenced a couple of them that I find great. Repeating the same content doesn't offer much value. :) And I wrote the entire content myself.
I don't think it's particularly helpful to dive into transformer models, positional encodings and self-attention at the very start of an introduction to LLMs. Understanding how those works does little to help explain what LLMs can do and how you can use them. I tend to stick with the higher level explanation that they can predict the next word (or next sentence) based on their training text, and then emphasize that w…
I'm not sure it's even helpful to talk about transformer at all[0]. Modern LLMs are different than the pure transformer architecture in many detailed aspects. The elephant in the room being that they're not even full transformer—only the decoder part. [0]: One reason: Never once did I need to know the transformer architecture in order to be able to use these models (prompt engineering, chaining, working with local mo…
> GPT-4 boasts 100 trillion parameters Source? I’ve never heard this. I’ve heard 1 trillion and that it’s a 8 x 175B ensemble.