Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OpenAI, Google and Anthropic are struggling to build more advanced AI

bloomberg.com

Post reply on HN
Print view View original

501–510 of 622 posts

Re: OpenAI, Google and Anthropic are struggling to build more advanced AI

#501

Post by n_ary »

tick_tock_tick wrote: post #392
sincerecook wrote: post #165

Earlier quoted context omitted.

> That's full multi-modal training with embodied agents (aka robots). 1x, Figure, Physical Intelligence, Tesla are all making rapid progress on functionality which is definitely beyond frontier LLMs because it is distinctly different. Cool, but we already have robots doing this in 2d space (aka self driving cars) that struggle not to kill people. How is adding a third dimension going to help? People are just refusing…

I ride in self driving cares basically once a week in SF (Waymo). It's always felt safer then a Uber and makes ways less risky maneuvers.

Could be because Uber or Taxi is trying to make most trips and maximize day earning while Waymo do not have that rush and can take things slow…

Of course Waymo needs money but if the car made fewer trips compared to Uber/Taxi, it is not suffering the same consequences.

We need to consider human factor and the severe lacking of that in these robot/self driving/LLM and drawing parallels is not a direction I am feeling comfortable.

End of the day, Tesla also sold half baked self drive that killed people, we should not forget.

Re: OpenAI, Google and Anthropic are struggling to build more advanced AI

#502

Post by gizajob »

BeefWellington wrote: post #204
drawnwren wrote: post #179

Earlier quoted context omitted.

Nvidia was not the best example. They get to moon in the case that any AI exponential hits. Most others have less of a wide probability distribution.

Yeah they're the shovel sellers of this particular goldrush. Most other businesses trying to actually use LLMs are the riskier ones, including OpenAI, IMO (though OpenAI is perhaps the least risky due to brand recognition).

I’d say it’s more about the fact that they make useful products rather than brand recognition.

Re: OpenAI, Google and Anthropic are struggling to build more advanced AI

#503

Post by Animats »

nickpsecurity wrote: post #470
Animats wrote: post #100

"While the model was initially expected to significantly surpass previous versions of the technology behind ChatGPT, it fell short in key areas, particularly in answering coding questions outside its training data." Right. If you generate some code with ChatGPT, and then try to find similar code on the web, you usually will. Search for unusual phrases in comments and for variable names. Often, something from Stack Ov…

The brain solves that problem. It seems to involve memory and specialized regions. I found a few groups building hippocampus-like, research models. One had content-addressable memory. There was another one that claimed to get rid of hallucinations. They also said it takes 50-100 epochs for regular architectures to actually memorize something. Their paper is below in case people qualified to review it want to. https:/…

Comments on that paper? PDF: [1]

What they are measuring, it seems, is whether LLMs can be built which will retrieve a reliable known correct answer on request. That's an information retrieval problem, and, in fact, they solve it by adding "Memory Experts" which are basically data storage.

It's not clear that this helps either replies which require synthesizing disparate information, or detecting that the training data does not contain info needed to construct a reply.

[1] https://arxiv.org/pdf/2406.17642

Re: OpenAI, Google and Anthropic are struggling to build more advanced AI

#504

Post by eslaught »

surrTurr wrote: post #498
osigurdson wrote: post #490

This "running out of data" thing suggests that there is something fundamentally wrong with how things are working. A new driver does not need to experience 8000 different rabbit-on-road situations from all angles to know to slow down when we see one on the road. Similarly we don't need 10,000 addition examples to learn how to add. It is as though there is no generalization in the models - just fundamentally search.

i think you underestimate the amount of data a driver experiences in a single 5 minute drive

I never get this argument.

I've seen a deer on a road maybe once. I've seen a rabbit on a road zero times. But I know what to do if I see one.

Is that because the "video" of my perception has many "frames"? Even if that's true at some level, I think it's massively missing the point. Yeah, so I saw that one deer from a lot of angles. But current AI training is like the equivalent of taking every deer that has ever been on camera in the history of the human species.

Somehow I'm still dramatically better at generalization than the AI. Surely that's an algorithm difference.

Re: OpenAI, Google and Anthropic are struggling to build more advanced AI

#505

Post by youoy »

Jensson wrote: post #462
youoy wrote: post #170

Earlier quoted context omitted.

Don't get caught in the superficial analysis. They "understand" things. It is a fact that LLMs experience a phase transition during training, from positional information to semantic understanding. It may well be the case that with scale there is another phase transition from semantic to something more abstract that we identify more closely with reasoning. It would be an emergent property of a sufficiently complex sys…

They understand sentences but not words.

What do you mean by that? We have the monosemanticity results [0]

[0] https://transformer-circuits.pub/2024/scaling-monosemanticit...

Re: OpenAI, Google and Anthropic are struggling to build more advanced AI

#506

Post by malthaus »

LASR wrote: post #115

Question for the group here: do we honestly feel like we've exhausted the options for delivering value on top of the current generation of LLMs? I lead a team exploring cutting edge LLM applications and end-user features. It's my intuition from experience that we have a LONG way to go. GPT-4o / Claude 3.5 are the go-to models for my team. Every combination of technical investment + LLMs yields a new list of potential…

it's the equivalent of the "we overestimate the impact of technology in the short-term and underestimate the effect in the long run" quote.

everyone is looking at llm scores & strawberry gotchas while ignoring the trillions of market potential in replacing existing systems and (yes) people with the current capabilities. identifying the use cases, finetuning the models and (most importantly) actually rolling this out in existing organizations/processes/systems will be the challenge long before the base models' capabilities will be

it is worth working on those issues now and get the ball rolling, switching out your models for future more capable ones will be the easy part later on.

Re: OpenAI, Google and Anthropic are struggling to build more advanced AI

#507

Post by Earw0rm »

slashdave wrote: post #411
limaoscarjuliet wrote: post #211

Earlier quoted context omitted.

Scaling laws are not dead. The number of people predicting death of Moore's law doubles every two years. - Jim Keller https://www.youtube.com/live/oIG9ztQw2Gc?si=oaK2zjSBxq2N-zj1...

Moore's law is doomed. At some point you start reaching the level of individual atoms. This is just physics.

The limits are engineering, not physics. Atoms need not be a barrier for a long time if you can go fully 3D, for example, but manufacturing challenges, power and heat get in the way long before that.

Then you can go ultra-wide in terms of cores, dispatchers and vectors (essentially building bigger and bigger chips), but an algorithm which can't exploit that will be little faster on today's chips than on a 4790K from ten years ago.

Re: OpenAI, Google and Anthropic are struggling to build more advanced AI

#508

Post by soheil »

physicsguy wrote: post #488
soheil wrote: post #431

Earlier quoted context omitted.

How is self-driving a 2D problem when you navigate a 3D world? (please do visit hilly San Francisco sometime) not to mention additional dimensions like depth, velocity vectors among others.

The visual input and sensory input to the self driving function are of the 3D world but the car is still constrained to move along a 2D topological surface, it’s not moving up and down other than by following the curvature of that

So based on your argument they actually operate in 1D since roads go in one direction and lanes and intersections are constrained to a predetermined curly line.

Re: OpenAI, Google and Anthropic are struggling to build more advanced AI

#509

Post by datahack »

The next wave won’t be monolithic but network-driven. Orchestration has the potential to integrate diverse AI systems and complementary technologies, such as advanced fact-checking and rule-based output frameworks.

This methodological growth could make LLMs more reliable, consistent, and aligned with specific use cases.

The skepticism surrounding this vision mirrors early doubts about the early internet fairly concisely.

Initially, the internet was seen as fragmented collection of isolated systems without a clear structure or purpose. It really was. You would gopher somewhere and get a file, and eventually we had apps like like pine for email, but as cool as it was it has limited utility.

People doubted it could ever become the seamless, interconnected web we know today.

Yet, through protocols, shared standards, and robust frameworks, the internet evolved into a powerful network capable of handling diverse applications, data flows, and user needs.

In the same way, LLM orchestration will mature by standardizing interfaces, improving interoperability, and fostering cooperation among varied AI models and support systems.

Just as the internet needed HTTP, TCP/IP, and other protocols to unify disparate networks, orchestrated AI systems will require foundational frameworks and “rules of the road” that bring cohesion to diverse technologies.

We are at the veeeeery infancy of this era and have a LONG way to go here. Some of the progress looks clear and a linear progression, but a lot, like the Internet, will just take a while to mature and we shouldn’t forget what we learned the last time we faced a sea change technological revolution.

Post reply on HN