21–27 of 27 posts
> GPT-4 boasts 100 trillion parameters Source? I’ve never heard this. I’ve heard 1 trillion and that it’s a 8 x 175B ensemble.
OpenAI hasn't confirmed it yet, so as of now, there are no reliable sources to rely on for this information. I just removed that part.
Earlier quoted context omitted.
The entire article looks like an amalgam of stitched together pieces taken from different sources without much care.
Almost like an AI written article
Many of the linked references are too recent to be in the training corpus (Llama 2, for example), so unless there's some web-search component to this it looks like an LLM wrote the first draft, and a human went through to edit, add links, and populate with images, etc.
I don't think it's particularly helpful to dive into transformer models, positional encodings and self-attention at the very start of an introduction to LLMs. Understanding how those works does little to help explain what LLMs can do and how you can use them. I tend to stick with the higher level explanation that they can predict the next word (or next sentence) based on their training text, and then emphasize that w…
One application of casting things to dictionaries is that it makes it easy to grab keyterms from the ingested texts. With the right datastore, we can take the binary set objects and query them very quickly for similarity overlap of other texts, similar to how vector search might return a cosine distance (multi-dimensional angle offset?).
I was thinking these sets could be used to segment vector space, and the only thing that makes that possible is that LLMs are great to use for things like this.
Earlier quoted context omitted.
I'm not sure it's even helpful to talk about transformer at all[0]. Modern LLMs are different than the pure transformer architecture in many detailed aspects. The elephant in the room being that they're not even full transformer—only the decoder part. [0]: One reason: Never once did I need to know the transformer architecture in order to be able to use these models (prompt engineering, chaining, working with local mo…
>I think we need something that at least captures the general mechanism by which LLMs predict the next token. This bit varies a lot since the capabilities involved in prediction depend on the data. If the text is a math book and the prompt is "... three plus five apples is a total of " the crucial capability is arithmetic (plus of course NL capabilities). On the other hand, if you're completing a post from /r/relatio…
Q: if one basket contains 5 apples and another basket contains 3 plums, and another basket contains 2 walnuts how many fruits are there combined?
A: The first basket contains \*5 apples\*. The second basket contains \*3 plums\*. The third basket contains \*2 walnuts\*. Adding up the numbers of fruits in each basket, there are a total of \*5 + 3 + 2 = 10 fruits combined\*.
Q: is a walnut a fruit?
A: You're right, a walnut is not technically a fruit. A walnut is a type of tree nut. To correctly answer the original question:
The first basket contains 5 apples. The second basket contains 3 plums. While the third basket contains 2 walnuts, walnuts are not fruits. So the total number of fruits combined is 5 apples + 3 plums = 8 fruits.Earlier quoted context omitted.
>I think we need something that at least captures the general mechanism by which LLMs predict the next token. This bit varies a lot since the capabilities involved in prediction depend on the data. If the text is a math book and the prompt is "... three plus five apples is a total of " the crucial capability is arithmetic (plus of course NL capabilities). On the other hand, if you're completing a post from /r/relatio…
The way LLMs process text is a lot like a typical human (top-of-the-head answer vs actually analyzing the question in depth). This is a recurring pattern with the current LLMs I've interacted with (ex., A: "x" Q: "but what about y?" A: "oh, you're right, now that I [actually] think about it, really z"). Example (Anthropic's Claude): Q: if one basket contains 5 apples and another basket contains 3 plums, and another b…
Of course, if we'd want to generalize it, we should have a structure where the model itself can actually make the decision to continue work for one more iteration, and also ensure that it can retain some of the temporary work-in-progress notes (currently it can see only what the previous iteration output as part of the answer).
Earlier quoted context omitted.
The way LLMs process text is a lot like a typical human (top-of-the-head answer vs actually analyzing the question in depth). This is a recurring pattern with the current LLMs I've interacted with (ex., A: "x" Q: "but what about y?" A: "oh, you're right, now that I [actually] think about it, really z"). Example (Anthropic's Claude): Q: if one basket contains 5 apples and another basket contains 3 plums, and another b…
An interesting fundamental limitation of transformer LLMs is that their architecture doesn't allow for the option for the model to choose to think a bit more or to choose to review their answer, as the quantity and structure computations they'll do is fixed in the architecture without any conditionals - but you repeating the same prompt means that the model gets to run a second iteration while having the result of th…
"Think before you speak: Training Language Models With Pause Tokens"
Basic idea is to teach the LLM to occasionally insert a "pause" token, which outputs nothing but gives it a chance to perform another round of operations on the way to the answer.
Earlier quoted context omitted.
any recommendations for explainers that operate on this level of abstraction?
Here's my most recent attempt at this: https://simonwillison.net/2023/Aug/27/wordcamp-llms/#what-th...