Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Many in the AI field think the bigger-is-better approach is running out of road

economist.com

Post reply on HN
Print view View original

41–50 of 354 posts

Re: Many in the AI field think the bigger-is-better approach is running out of road

#41

Post by sgt101 »

nextworddev wrote: post #14
whimsicalism wrote: post #8

Data requirements are overstated - you can train on longer and longer sequences and I am pretty sure most organizations are still using the “show the model the data only once“ approach which is just wasteful. Compute challenges are more real, but we are seeing for the first time huge amounts of global capital being allocated to solve specifically these problems, so I am curious what fruit that will bear in a few year…

Yep. We are in the very early innings of capital being deployed to all this.

Ok - what's the ROI on the $10bn (++) that OpenAI have had?

So far I reckon This isn't what VC's (or microsoft) dream of.

Re: Many in the AI field think the bigger-is-better approach is running out of road

#42

Post by TeMPOraL »

Animats wrote: post #11

We need a way to make tight little specialist models that don't hallucinate and reliably report when they don't know. Trying to cram all of the web into a LLM is a dead end.

How much general "thinking"[0] would you want those "tight little specialist models" to retain? I think that cramming "all of the web" is actually crucial for this capability[1], so at least with LLM-style models, you likely can't avoid it. The text in the training data set doesn't encode just the object-level knowledge, but indirectly also higher-level, cross-domain and general concepts; cutting down on the size and breadth of the training data may cause the network to lose the ability to "understand"[0].

--

[0] - Or "something very convincingly pretending to think by parroting stuff back", if you're closer to the "stochastic parrot" view.

[1] - Per my hand-wavy hypothesis that the bulk of what we call thinking boils down to proximity search in extremely high-dimensional space.

Re: Many in the AI field think the bigger-is-better approach is running out of road

#43

Post by nextworddev »

sgt101 wrote: post #41
nextworddev wrote: post #14

Earlier quoted context omitted.

Yep. We are in the very early innings of capital being deployed to all this.

Ok - what's the ROI on the $10bn (++) that OpenAI have had? So far I reckon This isn't what VC's (or microsoft) dream of.

I think Azure OpenAI service is growing at 1000% per quarter according to last earnings call

Re: Many in the AI field think the bigger-is-better approach is running out of road

#44

Post by krona »

satellite2 wrote: post #23

How is the economist qualified to answer this question?

Usually its the most credentialed I'm least interested in listening to, especially when the value of those credentials are dependant upon the future looking a particular way.

Re: Many in the AI field think the bigger-is-better approach is running out of road

#45

Post by simion314 »

TX81Z wrote: post #32
Animats wrote: post #11

We need a way to make tight little specialist models that don't hallucinate and reliably report when they don't know. Trying to cram all of the web into a LLM is a dead end.

GPT4 is really good at code and you can generally verify hallucination easily. The other good use cases are using LLM to turn natural language prompts into API calls to real data.

>GPT4 is really good at code

for popular languages, though for JS it most of the time outputs obsolete syntax and code.

Today i tried to do a bit of scripting with my son in Garrys Mod, it uses Expression 2 for a Wiremod module. GPT hallucinated a lot of functions and the worst part it switched almost each time from e2 to lua.

It is good at solving homeworks for students, or solving popular problems in popular languages and libraries though it might give you an ugly solution and ugly code, it is probably trained on bad code too and it did not learn to prefer good code over bad code.

Re: Many in the AI field think the bigger-is-better approach is running out of road

#46

Post by semiquaver »

satellite2 wrote: post #23

How is the economist qualified to answer this question?

Not sure where I heard this but it’s apparently a common trope that many of the politicians and leaders that treat The Economist as close to holy writ are often horrified to learn that most the staff is actually a bunch of very precocious 20-somethings that are good at research and writing in an authoritative tone.

Actually, now that I think of it, not so different from LLMs…

(Full disclosure, I’ve been a subscriber for a couple of decades)

Re: Many in the AI field think the bigger-is-better approach is running out of road

#47

Post by Forgeties79 »

stavros wrote: post #29
JimtheCoder wrote: post #22

Earlier quoted context omitted.

So, search isn't dead after all...

Search has definitely been dead since before LLMs, we just don't have a replacement yet.

For a while my replacement was “use google, add ‘reddit’ at the end.” Not sure how much longer that will work given even just this limited blackout impacted how effective that was lol

Re: Many in the AI field think the bigger-is-better approach is running out of road

#48

Post by samstave »

rco8786 wrote: post #19
Animats wrote: post #11

We need a way to make tight little specialist models that don't hallucinate and reliably report when they don't know. Trying to cram all of the web into a LLM is a dead end.

Bingo. I've been beating this drum since the initial GPT-3 awe.. The future of AI is bespoke, purpose-driven models trained on a combination of public and (importantly) proprietary data. Data is still king.

It will be a terrific moment when we can browse a galaxy map or other lists of LLMs, and select "context" groups - so you can say from history, give me X

Re: Many in the AI field think the bigger-is-better approach is running out of road

#49

Post by nomel »

Animats wrote: post #11

We need a way to make tight little specialist models that don't hallucinate and reliably report when they don't know. Trying to cram all of the web into a LLM is a dead end.

> that don't hallucinate

“Hallucination” is part of thought. Solving a new problem requires hallucinating new, non existing, possible outcomes and solutions, to find one that will work. It seems that eliminating the ability to interpolate and extrapolate (hallucinations) would make intelligence impossible. It would eliminate creativity, tying together new concepts, creation, etc.

Is the goal AI, or a nice database front end, to reference facts? Is intelligence facts, or is it the flexibility and the ability to handle and create the novel, things that are new?

The ability to have confidence, and know and respond to it, seems important, but that’s surely different than the elimination of hallucinations.

I’m probably misunderstanding something, and/or don’t know what I’m talking about.

Re: Many in the AI field think the bigger-is-better approach is running out of road

#50

Post by thelittleone »

samstave wrote: post #48
rco8786 wrote: post #19

Earlier quoted context omitted.

Bingo. I've been beating this drum since the initial GPT-3 awe.. The future of AI is bespoke, purpose-driven models trained on a combination of public and (importantly) proprietary data. Data is still king.

It will be a terrific moment when we can browse a galaxy map or other lists of LLMs, and select "context" groups - so you can say from history, give me X

That doesn't sound ideal at all to me. It also sounds like a bottleneck. I want to ask AI once and it recommends or selects the best "context" group.
Post reply on HN