If GPT-5 had passed the A/B testing OpenAI likes to do, it would have been released already. Instead, it seems they are clearly concerned the audience would not find it superior enough to GPT-4. So, the bluff must go on until the right cards appear.
OpenAI, Google and Anthropic are struggling to build more advanced AI
481–490 of 622 posts
Re: OpenAI, Google and Anthropic are struggling to build more advanced AI
#482Earlier quoted context omitted.
I don't think we've even started to get the most value out of current gen LLMs. For starters very few people are even looking at sampling which is a major part of the model performance. The theory behind these models so aggressively lags the engineering that I suspect there are many major improvements to be found just by understanding a bit more about what these models are really doing and making re-designs based on…
The reason people are holding out is that the current generation of models are still pretty poor in many areas. You can have it craft an email, or to review your email, but I wouldn't trust an LLM with anything mission-critical. The accuracy of the generated output is too low be trusted in most practical applications.
Google (even now) wasn't absolutely accurate either. That didn't stop it from becoming many billions worth.
> You can have it craft an email, or to review your email, but I wouldn't trust an LLM with anything mission-critical
My point is that an entire world lies between these two extremes.
Re: OpenAI, Google and Anthropic are struggling to build more advanced AI
#483Earlier quoted context omitted.
> Right. If you generate some code with ChatGPT, and then try to find similar code on the web, you usually will. People who "follow" AI, as the latest fad they want to comment on and appear intelligent about, repeat things like this constantly, even though they're not actually true for anything but the most trivial hello-world types of problems. I write code all day every day. I use Copilot and the like all day every…
You’re solving novel problems all day every day?
If we weren't, we (as in developers) weren't needed, right?
Re: OpenAI, Google and Anthropic are struggling to build more advanced AI
#484Earlier quoted context omitted.
Moore's law is doomed. At some point you start reaching the level of individual atoms. This is just physics.
You are missing the economic component.. it isn't just how small can a transistor be.. it was really about how many transistors can you get for your money. So even when we reach terminal density, we probably haven't reached terminal economics.
I didn't say we have currently reached a limit. I am saying that there obvious is a limit (at some point). So, scaling cannot go forever. This is a counterpoint to the dubious analogy with deep learning.
Re: OpenAI, Google and Anthropic are struggling to build more advanced AI
#485The biggest weakness of generative AI to me is knowledge. It gives the impression of knowledge about the world without actually having a model of the world or any sense of what it does or does not know.
For example recently I asked it to generate some phrases for a list of words, along with synonym and antonym lists.
The phrases were generally correct and appropriate (some mistakes but that’s fine). The synonyms/antonyms were misaligned to the list (so strictly speaking all wrong) and were often incorrect anyway. I imagine it would be the same if you asked for definitions of a list of words.
If you ask it to correct it just generates something else which is often also wrong. It’s certainly superficially convincing in many domains but once you try to get it to do real work it’s wrong in subtle ways.
Re: OpenAI, Google and Anthropic are struggling to build more advanced AI
#486Question for the group here: do we honestly feel like we've exhausted the options for delivering value on top of the current generation of LLMs? I lead a team exploring cutting edge LLM applications and end-user features. It's my intuition from experience that we have a LONG way to go. GPT-4o / Claude 3.5 are the go-to models for my team. Every combination of technical investment + LLMs yields a new list of potential…
> potential applications > if you ... > for example ... Yes there seems to be lots of potential. Yes we can brainstorm things that should work. Yes there is a lot of examples of incredible things in isolation. But it's a little bit like those youtube videos showing amazing basketball shots in 1 try, when in reality lots of failed attempts happened beforehand. Except our users experience the failed attempts (LLM repli…
We’ve found that the text it generates in our RAG application is good, but it cocks up probably 5-10% of the time doing the inline references to the documents which users think is a bug and which we aren’t able to fix. This is static rather than interactively generated too
Re: OpenAI, Google and Anthropic are struggling to build more advanced AI
#487Earlier quoted context omitted.
I don't think we've even started to get the most value out of current gen LLMs. For starters very few people are even looking at sampling which is a major part of the model performance. The theory behind these models so aggressively lags the engineering that I suspect there are many major improvements to be found just by understanding a bit more about what these models are really doing and making re-designs based on…
Great & motivational comment. Any pointers on where to start playing with the internals and sampling? Doesn’t need to be comprehensive, I just don’t know where to jump off from.
Afaiu “sampling” here, it is controlled with (not only?) topk and temp parameters in e.g. “text generation web ui”. You may find these in other frontends probably too.
This ofc implies local models and that you have a decent cpu + min 64gb of ram to run above 7b-sized model.
https://github.com/oobabooga/text-generation-webui
https://huggingface.co/models?pipeline_tag=text-generation&s...
Re: OpenAI, Google and Anthropic are struggling to build more advanced AI
#488Earlier quoted context omitted.
> That's full multi-modal training with embodied agents (aka robots). 1x, Figure, Physical Intelligence, Tesla are all making rapid progress on functionality which is definitely beyond frontier LLMs because it is distinctly different. Cool, but we already have robots doing this in 2d space (aka self driving cars) that struggle not to kill people. How is adding a third dimension going to help? People are just refusing…
How is self-driving a 2D problem when you navigate a 3D world? (please do visit hilly San Francisco sometime) not to mention additional dimensions like depth, velocity vectors among others.
The visual input and sensory input to the self driving function are of the 3D world but the car is still constrained to move along a 2D topological surface, it’s not moving up and down other than by following the curvature of that
Re: OpenAI, Google and Anthropic are struggling to build more advanced AI
#489Question for the group here: do we honestly feel like we've exhausted the options for delivering value on top of the current generation of LLMs? I lead a team exploring cutting edge LLM applications and end-user features. It's my intuition from experience that we have a LONG way to go. GPT-4o / Claude 3.5 are the go-to models for my team. Every combination of technical investment + LLMs yields a new list of potential…
Beyond just RAG, I'm fairly bullish on finetuning. For example, Qwen2.5-Coder-32B-Instruct is much better than Qwen2.5-72B-Instruct at coding... Despite simply being a smaller version of the same model, finetuned on code. It's on par with Sonnet 3.5 and 4o on most benchmarks, whereas the simple chat-tuned 72B model is much weaker.
And while Qwen2.5-Coder-32B-Instruct is a pretty advanced finetune — it was trained on an extra 5 trillion tokens — even smaller finetunes have done really well. For example, Dracarys-72B, which was a simpler finetune of Qwen2.5-72B using a modified version of DPO on a handmade set of answers to GSM8K, ARC, and HellaSwag, significantly outperforms the base Qwen2.5-72B model on the aider coding benchmarks.
There's a lot of intelligence we're leaving on the floor, because everyone is just prompting generic chat-tuned models! If you tune it to do something else, it'll be really good at the something else.
Re: OpenAI, Google and Anthropic are struggling to build more advanced AI
#490This "running out of data" thing suggests that there is something fundamentally wrong with how things are working. A new driver does not need to experience 8000 different rabbit-on-road situations from all angles to know to slow down when we see one on the road. Similarly we don't need 10,000 addition examples to learn how to add. It is as though there is no generalization in the models - just fundamentally search.