111–120 of 205 posts
This just confirms I'm not minimally competent in this conversation. Is there a "college freshman" explainer? GPT seems to be doing something incredibly different than prior AI. Is it really a Bayesian "next word" chooser at incredible scale?
Gpt is not a bayesian next word chooser. It does something different.
Can anyone explain why the GPT response to the same question 4 days ago is different from today's: [ 4 days ago] > Can I learn to sing like Pavarotti? It is possible to learn to sing like Pavarotti, but it will require a lot of practice, dedication, and training. Pavarotti was a renowned opera singer known for his incredible vocal range, powerful voice, and technical mastery. He had many years of training and experie…
This just confirms I'm not minimally competent in this conversation. Is there a "college freshman" explainer? GPT seems to be doing something incredibly different than prior AI. Is it really a Bayesian "next word" chooser at incredible scale?
You're looking for the paper 'attention is all you need'. Gpt is not a bayesian next word chooser. It does something different.
Earlier quoted context omitted.
The problem with ChatGPT's "knowledge" is that it isn't trustworthy. It will happily output very confident sounding nonsense, or blatantly incorrect statements. We need a way to verify how accurate it's outputs are
ChatGPT to be employed in marketing positions immediately.
Earlier quoted context omitted.
Current GPT version is not very good at math, and it's not at all clear whether this can be fixed within a language model. Q: I have 2 cats and 1 horse. How many legs do they have in total? A: If you have two cats and one horse, then the total number of legs they have is 8. Cats typically have four legs, while horses have four legs as well. So two cats would have a total of 8 legs, and when you add the four legs of t…
These errors are common and important to note. Though what I find fascinating is how often I can prompt engineer around these errors by telling GPT to just be smarter and think more carefully. Just tried the following for instance. Q: You are a master logician, who can solve all manner of language problems and have excellent math skills. I have 2 cats and 1 horse. How many legs do they have in total? Show your reason…
Also, it looks to me like a language model is capable of reasoning if you let it execute a few times.
First, have it generate multiple outputs using different primings. Then have it choose it's favorite output.
Map, Reduce
Amazing insight, particularly section 6. "- The two important but different abilities of GPT-3.5 are *knowledge* and *reasoning*. Generally, it would be ideal if we could *offload the knowledge part to the outside retrieval system and let the language model only focus on reasoning.* This is because: - The model’s internal knowledge is always cut off at a certain time. The model always needs up-to-date knowledge to an…
It's unclear to me how you could separate knowledge and reasoning: - Reasoning typically requires base knowledge to work from. A side effect of training reasoning is embedding knowledge into the model parameters. - Even if you offload the search portion (either through outputting special tokens that are postprocessed, or applying the model in multiple steps with postprocessing), you still need embedded knowledge for…
After a bit of prompt engineering the model could query inventory, "manufacture" various recipes, and store the end products in inventory.
It might be possible to look at the weight activations as it reasons through contacting the external system over the emulated communication bus? For a suitably varied set of commands you might be able to find a subset of weights that are most correlated to the task and prune the others. Then you'd be left with a model that can retrieve and store information, as well as perform reasoning tasks.
Still has problems with working memory (the input token limit, since the model is auto-regressive) given all the external information is coming back in via the prompt, but ChatGPT seems to handle that gracefully right now.
Earlier quoted context omitted.
Competitors will remove the political correctness filters and become more prominent. A Chinese stable diffusion art site is already gaining traction, no filters.
No filters you say? That sounds terrible. Could you provide a link so that I can definitely avoid it.
Earlier quoted context omitted.
One float per param, so naively 175*4 = ~700GB on disk. Most recent models are trained in FP16 or BF16 so 350GB. And there's some work on quantizing them to INT8 so knock that down to a mere 175GB. You can definitely run it on a desktop computer using RAM and NVME offload to make up for the fact that you probably don't have 175GB of GPU memory available, but it won't be fast: https://huggingface.co/blog/bloom-inferen…
It'd be pretty surprising if you could quantize a text model and have it still work. It has to be using those lower bits to store text; it's not like you can round a letter up or down.
Earlier quoted context omitted.
It would have way more than 4 people in the search results though. GP said there's at least 4 because there's him, his wife and his kids. Even knowing Obama is a person is a knowledge-based leap. (To us humans) it's obvious the question means Barack Obama because he's the most notable subject for that name. But how do you prevent your AI from responding that the "Obama JS library is 5 years old" https://github.com/rg…
If it's based on frequency of training data, the ex-president will have far more hits in the training corpus. Now, on less talked about topics it doesn't sound any different than what happens with people Q "How old is Tim?" A "Which Tim are you talking about, you didn't give me crap to work with?"