Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OpenAI, Google and Anthropic are struggling to build more advanced AI

bloomberg.com

Post reply on HN
Print view View original

421–430 of 622 posts

Re: OpenAI, Google and Anthropic are struggling to build more advanced AI

#421

Post by dr_kiszonka »

crystal_revenge wrote: post #127
LASR wrote: post #115

Question for the group here: do we honestly feel like we've exhausted the options for delivering value on top of the current generation of LLMs? I lead a team exploring cutting edge LLM applications and end-user features. It's my intuition from experience that we have a LONG way to go. GPT-4o / Claude 3.5 are the go-to models for my team. Every combination of technical investment + LLMs yields a new list of potential…

I don't think we've even started to get the most value out of current gen LLMs. For starters very few people are even looking at sampling which is a major part of the model performance. The theory behind these models so aggressively lags the engineering that I suspect there are many major improvements to be found just by understanding a bit more about what these models are really doing and making re-designs based on…

Would you have any suggestions on how to play with the internals of these open models? I don't understand LLMs well, and would love to spend some experimenting, but I don't know where to start. Are any projects more appropriate for neophytes?

Re: OpenAI, Google and Anthropic are struggling to build more advanced AI

#422

Post by SpicyLemonZest »

mvdtnz wrote: post #311
SpicyLemonZest wrote: post #48

Earlier quoted context omitted.

I don't understand why you'd be so dismissive about this. It's looking less likely that it'll end up happening, but is it any less believable than getting general intelligence by training a blob of meat?

I feel like accusing people of being "so dismissive" was strongly associated with NFTs and cryptocurrency a few years ago, and now it's widely deployed against anyone skeptical of very expensive, not very good word generators.

I'm not sure what point you're making. It's true that people, including myself, were dismissive of cryptocurrency a few years ago; I think it's clear at this point that we were wrong, and it's not actually the case that the industry is a Ponzi scheme propped up by scammers like FTX.

Re: OpenAI, Google and Anthropic are struggling to build more advanced AI

#423

Post by woopwoop »

jmward01 wrote: post #175

Every negative headline I see about AI hitting a wall or being over-hyped makes me think of the early 2000's with that new thing the 'internet' (yes, I know the internet is a lot older than that). There is little doubt in my mind that ten years from now nearly every aspect of life will be deeply connected to AI just like the internet took over everything in the late 90's and early 2000's and is now deeply connected t…

That's funny, because to me these headlines about how deep learning is over-hyped and hitting the wall remind me of headlines from ten years ago about how... deep learning is over-hyped and hitting the wall.

Re: OpenAI, Google and Anthropic are struggling to build more advanced AI

#424

Post by summerlight »

I guess this is somewhat expected? The current frontier models probably already have exhausted most of the entropy in the training data accumulated over decades and the new training data is very sparse. And the current mainstream architectures are not capable of sophisticated searching and planning, essential aspects for generating new entropy out of thin air. o1 was an interesting attempt to tackle this problem, but we probably still have a long way to go.

Re: OpenAI, Google and Anthropic are struggling to build more advanced AI

#425

Post by kian »

cruffle_duffle wrote: post #301
machiaweliczny wrote: post #224

Earlier quoted context omitted.

Long context is a scam. Claude is best but it’s still gets lost with longer context

In my experience, the reality of long context windows doesn’t live up to the hype. When you’re iterating on something, whether it's code, text, or any document, you end up with multiple versions layered in the context. Every time you revise, those earlier versions stick around, even though only the latest one is the "most correct". What gets pushed out isn’t the last version of the document itself (since it’s FIFO),…

This is why I exclusively use the API to 'chat' with GPT -- complete control over the context presented.

Re: OpenAI, Google and Anthropic are struggling to build more advanced AI

#426

Post by kenjackson »

JohnMakin wrote: post #349
kenjackson wrote: post #299

Earlier quoted context omitted.

> because we understand the rough biological processes that cause this We don't have a rough understanding of the biological processes that cause this, unless you literally mean just the biological process and not how it actual impacts learning/intelligence. There's no evidence that we (brains) have achieved AGI, unless you tautologically define AGI as our brains.

> We don't have a rough understanding of the biological processes that cause this, Yes we do. We know how neurons communicate, we know how they are formed, we have great evidence and clues as to how this evolved and how our various neurological symptoms are able to interact with the world. Is it a fully solved problem? no. > unless you literally mean just the biological process and not how it actual impacts learning/…

> Yes we do. We know how neurons communicate, we know how they are formed, we have great evidence and clues as to how this evolved and how our various neurological symptoms are able to interact with the world. Is it a fully solved problem? no.

It's not even close to fully solved. We're still figuring out basic things like the purpose of dreams. We don't understand how memories are encoded or even things like how we process basic emotions like happiness. We're way closer to understanding LLMs than we are the brain, and we don't understand LLMs all that well still either. For example, look at the Golden Gate Bridge work for LLMs -- we have no equivalent for brains today. We've done much more advanced introspection work on LLMs in this short amount of time than we've done on the human brain.

Re: OpenAI, Google and Anthropic are struggling to build more advanced AI

#427

Post by airstrike »

iandanforth wrote: post #47

A few important things to remember here: The best engineering minds have been focused on scaling transformer pre and post training for the last three years because they had good reason to believe it would work, and it has up until now. Progress has been measured against benchmarks which are / were largely solvable with scale. There is another emerging paradigm which is still small(er) scale but showing remarkable res…

The gap from the virtual world of software and the brutally uncompromising nature of physical reality is wider than most people seem to accept.

It's almost like saying "we've already visited every place on Earth, surely Mars is just around the corner now"

Re: OpenAI, Google and Anthropic are struggling to build more advanced AI

#430

Post by torguyvg46787 »

rafaelmn wrote: post #277
iandanforth wrote: post #47

A few important things to remember here: The best engineering minds have been focused on scaling transformer pre and post training for the last three years because they had good reason to believe it would work, and it has up until now. Progress has been measured against benchmarks which are / were largely solvable with scale. There is another emerging paradigm which is still small(er) scale but showing remarkable res…

>There is another emerging paradigm which is still small(er) scale but showing remarkable results. That's full multi-modal training with embodied agents (aka robots). 1x, Figure, Physical Intelligence, Tesla are all making rapid progress on functionality which is definitely beyond frontier LLMs because it is distinctly different. Tesla is selling this view for almost a decade now in self-driving - how their car fleet…

The approaches are very limited, and it's essentially artificial artificial AI (and need a lot of human teleop demos).

At CoRL last week, the progress has noticeably plateaued. Roboticists notably were pessimistic that scaling laws will apply to robotics because of the embodiment issues.

Post reply on HN