71–80 of 354 posts
I remember how hyped people were seeing the progress from GPT3.5 to GPT4, people really felt like many jobs were going to be replaced very soon. The next big advancement was around the corner. I think the limitations of LLMs should be more salient to them by now.
We need a way to make tight little specialist models that don't hallucinate and reliably report when they don't know. Trying to cram all of the web into a LLM is a dead end.
How much general "thinking"[0] would you want those "tight little specialist models" to retain? I think that cramming "all of the web" is actually crucial for this capability[1], so at least with LLM-style models, you likely can't avoid it. The text in the training data set doesn't encode just the object-level knowledge, but indirectly also higher-level, cross-domain and general concepts; cutting down on the size and…
Even GPT 3.5 has trouble following instructions, but I’ve found that GPT 4 is almost flawless. I can tell it the document uses Australian English but to preserve US spelling for product names and it’ll do it!
One quirk is that it’s almost too good at following instructions. You have to tell it to preserve product names, vendors names, place names, etc… otherwise it’ll “correct” the spelling of anything you forgot to list.
We need a way to make tight little specialist models that don't hallucinate and reliably report when they don't know. Trying to cram all of the web into a LLM is a dead end.
How much general "thinking"[0] would you want those "tight little specialist models" to retain? I think that cramming "all of the web" is actually crucial for this capability[1], so at least with LLM-style models, you likely can't avoid it. The text in the training data set doesn't encode just the object-level knowledge, but indirectly also higher-level, cross-domain and general concepts; cutting down on the size and…
General purpose models containing significant overlap between Project Gutenberg and Github are unnecessary and don't scale. Moby Dick has little to do with C++ unless you're creating art for novelty's sake. This is entirely speculative, but I'm convinced ChatGPT is faking the appearance of a single oracle while delegating requests to specialized models under the hood. It scales better and makes sense than trying to serve a 1T model to address everybody's banal questions.
Like, at its core, for people who only want to write literature, give them a model with underweighed programming-related corpora. Writers don't need it, will never use it, and that space could be filled with training content relevant to literature. Anything else results in expensive, unscalable solutions or jack-of-all-trades, master-of-none outcomes.
In recent usage, GPT3.5 helped me hack my way through writing Pester tests for Powershell scripts for the first time, and I mean hack-- there were a lot of assumptions it made and things it got wrong. GPT4 did a much better job, but I couldn't help but think 3.5 probably has a ton of other training data in it that detracts from the specialization I needed from it in that context. For coding help, you don't want to ask some random librarian who occasionally recommends resources that don't exist; you ask someone who specializes in coding and trust they have familiarity with that domain.
How is the economist qualified to answer this question?
(Long-time Economist subscriber.)
Isn't the fundamental problem that LLM's don't actually understand anything (as greater concepts), but rather operate as complex probability machines? My 2 month active experience with ChatGPT-4 gave me the following takeaways: - when it's right, it's amazing; and when you, the operator, can recognize the niche use case where it performs really well, it can be a game-changer (although you could have programmed a tool…
But, things are defined by how they interact with the world around them.
A concept is its relations to other concepts.
Which does seem to be the general sort of thing that these models are trying to get at, even if they don't seem to do a great job of it.
Sam Altman has been saying this for months. Nothing noteworthy here for someone following the industry closely. A16z’s latest summary of the landscape was way more useful and relevant than this.
Isn't the fundamental problem that LLM's don't actually understand anything (as greater concepts), but rather operate as complex probability machines? My 2 month active experience with ChatGPT-4 gave me the following takeaways: - when it's right, it's amazing; and when you, the operator, can recognize the niche use case where it performs really well, it can be a game-changer (although you could have programmed a tool…
The next iteration will be trained on your own data where "when it's a little wrong, you (the expert) can fix the issue and move on without friction" so that case will become "when it's right" and some amount of "when it's any amount of wrong" cases will become "when it's a little wrong". A few more cycles of this and we could be looking at GPT-10 which is a complete replacement for most tasks.
Like you’d train the model to give you the most accurate response based on your current problem space, but when they changes, you’d have to retrain on what you’re working on but by that stage it’s already out of date ?
We need a way to make tight little specialist models that don't hallucinate and reliably report when they don't know. Trying to cram all of the web into a LLM is a dead end.
> that don't hallucinate “Hallucination” is part of thought. Solving a new problem requires hallucinating new, non existing, possible outcomes and solutions, to find one that will work. It seems that eliminating the ability to interpolate and extrapolate (hallucinations) would make intelligence impossible. It would eliminate creativity, tying together new concepts, creation, etc. Is the goal AI, or a nice database fr…
Any evidence to support this claim or just commentary ?
Sam Altman has been saying this for months. Nothing noteworthy here for someone following the industry closely. A16z’s latest summary of the landscape was way more useful and relevant than this.
Sam had been saying it but few people trust he is telling the truth imo. Maybe he was.