How does GPT obtain its ability? Tracing emergent abilities of language models
1–10 of 205 posts
Re: How does GPT obtain its ability? Tracing emergent abilities of language models
#2Amazing insight, particularly section 6.
"- The two important but different abilities of GPT-3.5 are *knowledge* and *reasoning*. Generally, it would be ideal if we could *offload the knowledge part to the outside retrieval system and let the language model only focus on reasoning.* This is because: - The model’s internal knowledge is always cut off at a certain time. The model always needs up-to-date knowledge to answer up-to-date questions. - Recall we have discussed that is 175B parameter is heavily used for storing knowledge. If we could offload knowledge to be outside the model, then the model parameter might be significantly reduced such that eventually, it can run on a cellphone (call this crazy here, but ChatGPT is already science fiction enough, who knows what the future will be)."
& "Yet there was a WebGPT paper published in Dec 2021. It is likely that this is already tested internally within OpenAI."
It definitely feels like this may be the next step in making this kind of system robust. It ends up being an interface for search.
Re: How does GPT obtain its ability? Tracing emergent abilities of language models
#3Imagine Wolfram's system with GPT integrated. I hope they're working on that.
Re: How does GPT obtain its ability? Tracing emergent abilities of language models
#4Amazing insight, particularly section 6. "- The two important but different abilities of GPT-3.5 are *knowledge* and *reasoning*. Generally, it would be ideal if we could *offload the knowledge part to the outside retrieval system and let the language model only focus on reasoning.* This is because: - The model’s internal knowledge is always cut off at a certain time. The model always needs up-to-date knowledge to an…
The problem with ChatGPT's "knowledge" is that it isn't trustworthy. It will happily output very confident sounding nonsense, or blatantly incorrect statements. We need a way to verify how accurate it's outputs are
Re: How does GPT obtain its ability? Tracing emergent abilities of language models
#5Amazing insight, particularly section 6. "- The two important but different abilities of GPT-3.5 are *knowledge* and *reasoning*. Generally, it would be ideal if we could *offload the knowledge part to the outside retrieval system and let the language model only focus on reasoning.* This is because: - The model’s internal knowledge is always cut off at a certain time. The model always needs up-to-date knowledge to an…
How much disk space does 175B parameters use? A float or half precision float per parameter or does it need pointers to connections too?
Given how responses are generated in seconds and for free I am fairly sure it could run on a desktop computer.
Re: How does GPT obtain its ability? Tracing emergent abilities of language models
#6Imagine Wolfram's system with GPT integrated. I hope they're working on that.
Current GPT version is not very good at math, and it's not at all clear whether this can be fixed within a language model.
Q: I have 2 cats and 1 horse. How many legs do they have in total?
A: If you have two cats and one horse, then the total number of legs they have is 8. Cats typically have four legs, while horses have four legs as well. So two cats would have a total of 8 legs, and when you add the four legs of the horse, that brings the total number of legs to 8.
Re: How does GPT obtain its ability? Tracing emergent abilities of language models
#7Amazing insight, particularly section 6. "- The two important but different abilities of GPT-3.5 are *knowledge* and *reasoning*. Generally, it would be ideal if we could *offload the knowledge part to the outside retrieval system and let the language model only focus on reasoning.* This is because: - The model’s internal knowledge is always cut off at a certain time. The model always needs up-to-date knowledge to an…
The problem with ChatGPT's "knowledge" is that it isn't trustworthy. It will happily output very confident sounding nonsense, or blatantly incorrect statements. We need a way to verify how accurate it's outputs are
so, much like other knowledge sources?
Re: How does GPT obtain its ability? Tracing emergent abilities of language models
#8Amazing insight, particularly section 6. "- The two important but different abilities of GPT-3.5 are *knowledge* and *reasoning*. Generally, it would be ideal if we could *offload the knowledge part to the outside retrieval system and let the language model only focus on reasoning.* This is because: - The model’s internal knowledge is always cut off at a certain time. The model always needs up-to-date knowledge to an…
How much disk space does 175B parameters use? A float or half precision float per parameter or does it need pointers to connections too? Given how responses are generated in seconds and for free I am fairly sure it could run on a desktop computer.
One float per param, so naively 175*4 = ~700GB on disk. Most recent models are trained in FP16 or BF16 so 350GB. And there's some work on quantizing them to INT8 so knock that down to a mere 175GB. You can definitely run it on a desktop computer using RAM and NVME offload to make up for the fact that you probably don't have 175GB of GPU memory available, but it won't be fast: https://huggingface.co/blog/bloom-inference-pytorch-scripts
OpenAI generates responses so fast by doing the generation in parallel across something like 8x80GB A100s (I don't know the exact details of their hardware setup, but NVIDIA's open FasterTransformer library achieves low latency for large models this way).
Re: How does GPT obtain its ability? Tracing emergent abilities of language models
#9Imagine Wolfram's system with GPT integrated. I hope they're working on that.
Current GPT version is not very good at math, and it's not at all clear whether this can be fixed within a language model. Q: I have 2 cats and 1 horse. How many legs do they have in total? A: If you have two cats and one horse, then the total number of legs they have is 8. Cats typically have four legs, while horses have four legs as well. So two cats would have a total of 8 legs, and when you add the four legs of t…
It would have got it right if you got it to show its working.
That's the trick to doing computations with LLMs, the computation per token is constant so you want to split the computation over many tokens.
> I have 2 cats and 1 horse. How many legs do they have in total? show your working.
> If you have 2 cats and 1 horse, they would have a total of 10 legs. Cats typically have 4 legs, so 2 cats would have 2 x 4 = >8 legs. Horses have 4 legs, so 1 horse would have 1 x 4 = >4 legs. Together, the 2 cats and 1 horse would have 8 + 4 = >12 legs.
Re: How does GPT obtain its ability? Tracing emergent abilities of language models
#10Imagine Wolfram's system with GPT integrated. I hope they're working on that.
Current GPT version is not very good at math, and it's not at all clear whether this can be fixed within a language model. Q: I have 2 cats and 1 horse. How many legs do they have in total? A: If you have two cats and one horse, then the total number of legs they have is 8. Cats typically have four legs, while horses have four legs as well. So two cats would have a total of 8 legs, and when you add the four legs of t…
It's very reactive to the prompt:
> i have 23 horses each with a normal number of legs, along with 14 dogs, each with a normal number of legs. how many legs in total do all of the animals have?
> Based on the information you provided, it sounds like you have 23 horses and each horse has 4 legs, and you have 14 dogs and each dog has 4 legs. If that is correct, then the total number of legs among all of the animals would be 23 x 4 + 14 x 4 = 92 + 56 = >148 legs.