Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GPT-4, without specialized training, beat a GPT-3.5 class model that cost $10M

threads.net

Post reply on HN
Print view View original

61–70 of 134 posts

Re: GPT-4, without specialized training, beat a GPT-3.5 class model that cost $10M

#61

Post by marviel »

minimaxir wrote: post #17

Extremely hot LLM take: You will often get better results with few-shot prompting (with good examples) on a modern LLM than with a finetuned LLM. Finetuning was the best option for weaker LLMs with lower context windows (e.g. the original GPT-3): both problems have been solved nowadays. The cost economics are much better with few-shot prompting to modern LLMs too: input tokens are super cheap (especially with the rec…

Several MSFT AI/ML friends actively dissuaded me and my team from fine-tuning. They said that it's pretty clear in all their internal tests that it "lobotomizes" the general reasoning capabilities of the model, unless you're really careful.

"All work and no play makes GPT a very dull AI"

Re: GPT-4, without specialized training, beat a GPT-3.5 class model that cost $10M

#62

Post by simonw »

rkagerer wrote: post #54
ldjkfkdsjnv wrote: post #22

Fine tuning will disappear, no reason to invest so heavily in it. Also, most LLM code being written right now (prompts, RAG, etc) will be obsolete once the next model comes out. Anyone starting an LLM application startup is arguably wasting their time, wait until the next iteration is out. Then you will know whats possible.

"Don't buy a computer today, because the faster one is coming out tomorrow"

Don't buy a computer today with a six month delivery lead time, because there's a company that releases computers with a same-day lead time with several improved models coming out next week.

Re: GPT-4, without specialized training, beat a GPT-3.5 class model that cost $10M

#63

Post by marviel »

MuffinFlavored wrote: post #4

Does anything currently beat GPT-4? I saw some comments here say to check out Claude. From what I can tell, Claude hasn't figure out yet how to do the whole "generate Python code and run it in a Juptyer notebook" for math yet.

Claude is excellent for brainstorming, being a thought partner, general knowledge acquisition tasks, and creative writing.

The one mixed-bag weak spot I've found is in coding -- It tends to make more "d'oh" mistakes while coding, but comes up with more creative solutions at the same time ¯\_(ツ)_/¯

Re: GPT-4, without specialized training, beat a GPT-3.5 class model that cost $10M

#64

Post by throwaway74432 »

LASR wrote: post #42

I lead AI teams at my company. I've advised leadership against any kind of training / fine-tuning anything. We're not in the business of training models. We will never be as good as OpenAI / Anthropic etc. Where the real value in applications is smarter prompting techniques and RAG. There is a lot of room at the bottom in doing "dumb" things and simply feeding models with the right context to deliver customer value.

Hear hear. I know a 3 person startup that has a "lead AI researcher" who is trying to train and fine-tune models. That's not their startup's purpose though... they have an actual product. So wtf are they doing? The lead AI guy thinks he's going to compete with these big companies and it's total fantasy.

LLMs are a commodity

Re: GPT-4, without specialized training, beat a GPT-3.5 class model that cost $10M

#65

Post by SgtBastard »

jnwatson wrote: post #49
LASR wrote: post #42

I lead AI teams at my company. I've advised leadership against any kind of training / fine-tuning anything. We're not in the business of training models. We will never be as good as OpenAI / Anthropic etc. Where the real value in applications is smarter prompting techniques and RAG. There is a lot of room at the bottom in doing "dumb" things and simply feeding models with the right context to deliver customer value.

It is trivial to fine tune these days. RAG is already irrelevant with large context windows.

A remarkable comment in that it is clear, confident and wrong.

Fine-tunes lead to catastrophic forgetting.

RAG is only irrelevant if you’re completely disinterested in cost and latency.

We also don’t have enough data to gauge performance of models >200k context window size when reasoning over inputs of that size, much of which will be irrelevant to any particular user. Multiple random needles in haystack tests work flawlessly, but rarely applies to real world activity.

Re: GPT-4, without specialized training, beat a GPT-3.5 class model that cost $10M

#66

Post by jorvi »

Solvency wrote: post #51
zapperdulchen wrote: post #40

Earlier quoted context omitted.

Seems like the bitter lesson is still right: http://www.incompleteideas.net/IncIdeas/BitterLesson.html

Whoa this guy says "computation" and not grammatically bastardized techbrospeak "compute" like some neckbeard equivalent of a caveman! For that alone I commend him.

Compute is.. I don’t know the exact English grammatical term but it’s like water. Computation is not.

“I have 1000 flops of compute” - works.

“I have 1000 flops of computation” - doesn’t work.

“That compute failed” - doesn’t work.

“That computation failed” - works.

They’re different.

Re: GPT-4, without specialized training, beat a GPT-3.5 class model that cost $10M

#68

Post by [deleted] »

Solvency wrote: post #51
zapperdulchen wrote: post #40

Earlier quoted context omitted.

Seems like the bitter lesson is still right: http://www.incompleteideas.net/IncIdeas/BitterLesson.html

Whoa this guy says "computation" and not grammatically bastardized techbrospeak "compute" like some neckbeard equivalent of a caveman! For that alone I commend him.

[deleted]

Re: GPT-4, without specialized training, beat a GPT-3.5 class model that cost $10M

#69

Post by qeternity »

smallnamespace wrote: post #50
thorum wrote: post #35

Earlier quoted context omitted.

That might be true for finetuning ChatGPT 3.5, but if you can finetune a small model (7B or less) to perform on par with GPT-4, while being faster and private, that’s a different story.

You definitely can't in the general case (for example, your 7B model is never going to be able to help much with coding, fine tuning or no). It can make sense if you have a particularly simple use case.

By definition you wouldn’t fine tune a 7B model to be generally as good at GPT4. You would just be trying to overfit some small amount of functionality in a narrow domain.

Re: GPT-4, without specialized training, beat a GPT-3.5 class model that cost $10M

#70

Post by nialse »

zapperdulchen wrote: post #40
minimaxir wrote: post #17

Extremely hot LLM take: You will often get better results with few-shot prompting (with good examples) on a modern LLM than with a finetuned LLM. Finetuning was the best option for weaker LLMs with lower context windows (e.g. the original GPT-3): both problems have been solved nowadays. The cost economics are much better with few-shot prompting to modern LLMs too: input tokens are super cheap (especially with the rec…

Seems like the bitter lesson is still right: http://www.incompleteideas.net/IncIdeas/BitterLesson.html

For those who were oblivious to it, like myself, the bitter lesson is written by Richard S Sutton who invented reinforcement learning a long, long time ago.
Post reply on HN