Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GPT-4, without specialized training, beat a GPT-3.5 class model that cost $10M

threads.net

Post reply on HN
Print view View original

71–80 of 134 posts

Re: GPT-4, without specialized training, beat a GPT-3.5 class model that cost $10M

#71

Post by qeternity »

throwaway74432 wrote: post #64
LASR wrote: post #42

I lead AI teams at my company. I've advised leadership against any kind of training / fine-tuning anything. We're not in the business of training models. We will never be as good as OpenAI / Anthropic etc. Where the real value in applications is smarter prompting techniques and RAG. There is a lot of room at the bottom in doing "dumb" things and simply feeding models with the right context to deliver customer value.

Hear hear. I know a 3 person startup that has a "lead AI researcher" who is trying to train and fine-tune models. That's not their startup's purpose though... they have an actual product. So wtf are they doing? The lead AI guy thinks he's going to compete with these big companies and it's total fantasy. LLMs are a commodity

That does indeed sound crazy. But finetuning is also a commodity these days. You can train a good Mistral LoRA in under 24 hours on a single consumer GPU. We’re talking about $10 of compute.

You can run a dozen of these LoRAs atop the same base model on the same infrastructure for a dozen specific use cases.

The inference quality, performance and cost can all be substantially better than GPT4 with prompting.

Re: GPT-4, without specialized training, beat a GPT-3.5 class model that cost $10M

#72

Post by [deleted] »

throwaway74432 wrote: post #64
LASR wrote: post #42

I lead AI teams at my company. I've advised leadership against any kind of training / fine-tuning anything. We're not in the business of training models. We will never be as good as OpenAI / Anthropic etc. Where the real value in applications is smarter prompting techniques and RAG. There is a lot of room at the bottom in doing "dumb" things and simply feeding models with the right context to deliver customer value.

Hear hear. I know a 3 person startup that has a "lead AI researcher" who is trying to train and fine-tune models. That's not their startup's purpose though... they have an actual product. So wtf are they doing? The lead AI guy thinks he's going to compete with these big companies and it's total fantasy. LLMs are a commodity

[deleted]

Re: GPT-4, without specialized training, beat a GPT-3.5 class model that cost $10M

#73

Post by thewakalix »

jorvi wrote: post #66
Solvency wrote: post #51

Earlier quoted context omitted.

Whoa this guy says "computation" and not grammatically bastardized techbrospeak "compute" like some neckbeard equivalent of a caveman! For that alone I commend him.

Compute is.. I don’t know the exact English grammatical term but it’s like water. Computation is not. “I have 1000 flops of compute” - works. “I have 1000 flops of computation” - doesn’t work. “That compute failed” - doesn’t work. “That computation failed” - works. They’re different.

https://en.wikipedia.org/wiki/Mass_noun

Re: GPT-4, without specialized training, beat a GPT-3.5 class model that cost $10M

#74

Post by jna_sh »

minimaxir wrote: post #17

Extremely hot LLM take: You will often get better results with few-shot prompting (with good examples) on a modern LLM than with a finetuned LLM. Finetuning was the best option for weaker LLMs with lower context windows (e.g. the original GPT-3): both problems have been solved nowadays. The cost economics are much better with few-shot prompting to modern LLMs too: input tokens are super cheap (especially with the rec…

“Modern” is an extremely funny delineation given the small temporal window of this whole thing

Re: GPT-4, without specialized training, beat a GPT-3.5 class model that cost $10M

#75

Post by simonw »

hallqv wrote: post #67

This discussion is so dumb - finetuning a base model costs ~$1 with LORA/QLORA and can yield same performance as gpt-4, but at 1/100 of the cost per token. What Bloomberg did for $10M was not finetuning..

"finetuning a base model costs ~$1 with LORA/QLORA and can yield same performance as gpt-4, but at 1/100 of the cost per token"

That's a big claim - can you back that up with any examples?

Re: GPT-4, without specialized training, beat a GPT-3.5 class model that cost $10M

#76

Post by Solvency »

jorvi wrote: post #66
Solvency wrote: post #51

Earlier quoted context omitted.

Whoa this guy says "computation" and not grammatically bastardized techbrospeak "compute" like some neckbeard equivalent of a caveman! For that alone I commend him.

Compute is.. I don’t know the exact English grammatical term but it’s like water. Computation is not. “I have 1000 flops of compute” - works. “I have 1000 flops of computation” - doesn’t work. “That compute failed” - doesn’t work. “That computation failed” - works. They’re different.

Literally not true. Compute is a verb. Computation is the right word in all of those cases. Or computational .

Re: GPT-4, without specialized training, beat a GPT-3.5 class model that cost $10M

#77

Post by VirusNewbie »

throwaway74432 wrote: post #64
LASR wrote: post #42

I lead AI teams at my company. I've advised leadership against any kind of training / fine-tuning anything. We're not in the business of training models. We will never be as good as OpenAI / Anthropic etc. Where the real value in applications is smarter prompting techniques and RAG. There is a lot of room at the bottom in doing "dumb" things and simply feeding models with the right context to deliver customer value.

Hear hear. I know a 3 person startup that has a "lead AI researcher" who is trying to train and fine-tune models. That's not their startup's purpose though... they have an actual product. So wtf are they doing? The lead AI guy thinks he's going to compete with these big companies and it's total fantasy. LLMs are a commodity

Doesn't it entirely depend on how specialized the training data for a given fine tuned model might be?

Re: GPT-4, without specialized training, beat a GPT-3.5 class model that cost $10M

#78

Post by krasin »

Finetuning LLMs is currently the most promising way for next-gen robotics. One of such works (PaLM-e) among other things measured the impact of finetuning on general purpose tasks: https://twitter.com/DannyDriess/status/1632904698108821505

In short, an 8B model could degrade almost 10x after being finetuned on robotics tasks, while 500B model experiences a very minor degradation (~4%) and there's a hope that with a larger-sized model, it could become either zero or even negative (adding more experience improves general purpose reasoning).

What I am saying is that while GPT-4 could beat a finetuned GPT-3.5 class model, I predict good things about finetuned GPT-4 class models, when they become practical outside of OpenAI/Google.

Re: GPT-4, without specialized training, beat a GPT-3.5 class model that cost $10M

#79

Post by FrustratedMonky »

marviel wrote: post #61
minimaxir wrote: post #17

Extremely hot LLM take: You will often get better results with few-shot prompting (with good examples) on a modern LLM than with a finetuned LLM. Finetuning was the best option for weaker LLMs with lower context windows (e.g. the original GPT-3): both problems have been solved nowadays. The cost economics are much better with few-shot prompting to modern LLMs too: input tokens are super cheap (especially with the rec…

Several MSFT AI/ML friends actively dissuaded me and my team from fine-tuning. They said that it's pretty clear in all their internal tests that it "lobotomizes" the general reasoning capabilities of the model, unless you're really careful. "All work and no play makes GPT a very dull AI"

"bitter lesson that building in how we think we think does not work in the long run"

Guess. Stop trying to shape the NN. And let it learn on its own.

Re: GPT-4, without specialized training, beat a GPT-3.5 class model that cost $10M

#80

Post by brianjking »

simonw wrote: post #55
thorum wrote: post #43

Earlier quoted context omitted.

I don’t know what the people who say Claude 3 is better than GPT-4 are using it for. It’s been consistently worse for everything I’ve thrown at it. Debugging a Python function this morning. Claude 3 Opus failed completely. GPT-4 found the bug, as well as two others I hadn’t even been looking for.

I've had the opposite experience: coding prompts that GPT-4 makes mistakes on Claude 3 Opus gets right the first time. As always, your results will vary based on your personal prompting style. My style apparently works great with Opus. Here's one example: GPT-4 gave me code that was missing some async/await keywords: https://chat.openai.com/share/117fb1ad-6361-41e2-be59-110f32... Claude 3 Opus with the same prompt go…

Yeah, Opus has entirely taken over any code specific use for me over ChatGPT 4 or OpenAI GPT-4 API.

Once Opus has the ability to run a code interpreter, it'll really be an exciting time.

Post reply on HN