61–70 of 354 posts
Earlier quoted context omitted.
It will be a terrific moment when we can browse a galaxy map or other lists of LLMs, and select "context" groups - so you can say from history, give me X
That doesn't sound ideal at all to me. It also sounds like a bottleneck. I want to ask AI once and it recommends or selects the best "context" group.
We need a way to make tight little specialist models that don't hallucinate and reliably report when they don't know. Trying to cram all of the web into a LLM is a dead end.
How much general "thinking"[0] would you want those "tight little specialist models" to retain? I think that cramming "all of the web" is actually crucial for this capability[1], so at least with LLM-style models, you likely can't avoid it. The text in the training data set doesn't encode just the object-level knowledge, but indirectly also higher-level, cross-domain and general concepts; cutting down on the size and…
My 2 month active experience with ChatGPT-4 gave me the following takeaways:
- when it's right, it's amazing; and when you, the operator, can recognize the niche use case where it performs really well, it can be a game-changer (although you could have programmed a tool to do the same narrow task)
- when it's a little wrong, you (the expert) can fix the issue and move on without friction
- when it's any amount of wrong and you are less than an expert, or specifically you are completely unfamiliar with the topic, you can waste an immense amount of time researching the output and/or iterating with the system to refine the result
Initially I thought it was a 10:1 ratio of performance to effort. But finally (as a developer) I settled on (1 to 1.5):1. It basically just changed the game for me from doing the actual hard work to working out how to tease the system into producing a reasonable result. And in the process (same as with co-pilot) I started to recognize how it was leading me to change my habits to avoid thinking and effort and instead rely on an external brain. If you could have a reliable external brain always available, that would be a fair trade. But when the external "brain" is unreliable and only available via certain interfaces, it's better to train yourself to be proactive, voracious (with respect to documentation), and tolerant of learning/producing cycles.
I once thought it would be a game-changer. Now I realize it is a game-changer, but in a similar way that offshoring was... it didn't improve or solve any problems, but it merely changed the work.
We need a way to make tight little specialist models that don't hallucinate and reliably report when they don't know. Trying to cram all of the web into a LLM is a dead end.
Bingo. I've been beating this drum since the initial GPT-3 awe.. The future of AI is bespoke, purpose-driven models trained on a combination of public and (importantly) proprietary data. Data is still king.
Earlier quoted context omitted.
For a while my replacement was “use google, add ‘reddit’ at the end.” Not sure how much longer that will work given even just this limited blackout impacted how effective that was lol
That hasn’t worked since about three months after companies found out people do it. It’s all astroturfing now days anyway and if it applies to products (which it for sure does) you can be sure that government actors caught on as well.
Isn't the fundamental problem that LLM's don't actually understand anything (as greater concepts), but rather operate as complex probability machines? My 2 month active experience with ChatGPT-4 gave me the following takeaways: - when it's right, it's amazing; and when you, the operator, can recognize the niche use case where it performs really well, it can be a game-changer (although you could have programmed a tool…
I also heard this. I unfortunately forget which study it was but yes, their paper spoke of likely diminishing returns at least at around 400-500B parameters for current LLM's. The recent news of GPT-4 running on 8x 220B LLM's (which doesn't equal a 8*220B size) fits that range and it's also questionable how much further we can push LLM's further by introducing multiple models like this, because this too eventually in…
Have those reports from George Hotz been confirmed? It seems plausible to me, but also suggests to me that we have further to go by using that parameter budget for depth rather than for width.
Earlier quoted context omitted.
> that don't hallucinate “Hallucination” is part of thought. Solving a new problem requires hallucinating new, non existing, possible outcomes and solutions, to find one that will work. It seems that eliminating the ability to interpolate and extrapolate (hallucinations) would make intelligence impossible. It would eliminate creativity, tying together new concepts, creation, etc. Is the goal AI, or a nice database fr…
>Is the goal AI, or a nice database front end, to reference facts? The latter given the kind of products that are currently being built with it. You don't want your code completion or news aggregator to hallucinate for the same reason you don't want your wrench to hallucinate, it's a tool. And as for hallucinations, that's a PR friendly misnomer for "it made **** up". Using the same phrase doesn't mean it has functio…
Obviously it's worth it to try and eliminate the incorrect information, but what grand-op is saying is we don't want to do that if it takes away some valuable emergent properties.
Earlier quoted context omitted.
Yep. We are in the very early innings of capital being deployed to all this.
Ok - what's the ROI on the $10bn (++) that OpenAI have had? So far I reckon This isn't what VC's (or microsoft) dream of.