Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Many in the AI field think the bigger-is-better approach is running out of road

economist.com

Post reply on HN
Print view View original

21–30 of 354 posts

Re: Many in the AI field think the bigger-is-better approach is running out of road

#21

Post by whimsicalism »

PartiallyTyped wrote: post #18
whimsicalism wrote: post #8

Data requirements are overstated - you can train on longer and longer sequences and I am pretty sure most organizations are still using the “show the model the data only once“ approach which is just wasteful. Compute challenges are more real, but we are seeing for the first time huge amounts of global capital being allocated to solve specifically these problems, so I am curious what fruit that will bear in a few year…

> using the “show the model the data only once“ approach which is just wasteful. According to the InstructGPT paper, that is not the case, showing the data multiple times results in overfitting.

1. You are just referring to fine tuning, I am referring to training the base model.

2. They still saw performance improvements which is why they did train on the data multiple times, you can see in the paper.

3. there was a recent paper demonstrating that reusing data still saw continued improvements in perplexity, i am on my ipad so cannot find it now

Re: Many in the AI field think the bigger-is-better approach is running out of road

#22

Post by JimtheCoder »

rco8786 wrote: post #19
Animats wrote: post #11

We need a way to make tight little specialist models that don't hallucinate and reliably report when they don't know. Trying to cram all of the web into a LLM is a dead end.

Bingo. I've been beating this drum since the initial GPT-3 awe.. The future of AI is bespoke, purpose-driven models trained on a combination of public and (importantly) proprietary data. Data is still king.

So, search isn't dead after all...

Re: Many in the AI field think the bigger-is-better approach is running out of road

#25

Post by jackmott42 »

I'm pretty excited by the possibilities. I am astounded by how much these language models can do with nothing but "predict the next word" as the core idea. I imagine in the near future having collections of a hundred different models, physics models, grammar models, fact models, sentiment models, vision models, wired all together by coordination models, and wired up to math tools and databases to ground truth when possible. I think it can get pretty wild.

Just chatGPT wired up to Wolfram Alpha is already pretty creepy amazing.

Re: Many in the AI field think the bigger-is-better approach is running out of road

#27

Post by causalmodels »

Animats wrote: post #11

We need a way to make tight little specialist models that don't hallucinate and reliably report when they don't know. Trying to cram all of the web into a LLM is a dead end.

It's not an either or. We're going to leverage the web trained LLMs to bootstrap the specialist models via combination of training token quality classifiers and synthetic data generation. Phi-1 is a pretty good example of this.

Re: Many in the AI field think the bigger-is-better approach is running out of road

#28

Post by xwdv »

I remember how hyped people were seeing the progress from GPT3.5 to GPT4, people really felt like many jobs were going to be replaced very soon. The next big advancement was around the corner. I think the limitations of LLMs should be more salient to them by now.

Re: Many in the AI field think the bigger-is-better approach is running out of road

#29

Post by stavros »

JimtheCoder wrote: post #22
rco8786 wrote: post #19

Earlier quoted context omitted.

Bingo. I've been beating this drum since the initial GPT-3 awe.. The future of AI is bespoke, purpose-driven models trained on a combination of public and (importantly) proprietary data. Data is still king.

So, search isn't dead after all...

Search has definitely been dead since before LLMs, we just don't have a replacement yet.

Re: Many in the AI field think the bigger-is-better approach is running out of road

#30

Post by JimtheCoder »

stavros wrote: post #29
JimtheCoder wrote: post #22

Earlier quoted context omitted.

So, search isn't dead after all...

Search has definitely been dead since before LLMs, we just don't have a replacement yet.

I mean more the concept of search, not the current implementation
Post reply on HN