Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GPT-4, without specialized training, beat a GPT-3.5 class model that cost $10M

threads.net

Post reply on HN
Print view View original

101–110 of 134 posts

Re: GPT-4, without specialized training, beat a GPT-3.5 class model that cost $10M

#101

Post by singularity2001 »

redox99 wrote: post #87
LASR wrote: post #42

I lead AI teams at my company. I've advised leadership against any kind of training / fine-tuning anything. We're not in the business of training models. We will never be as good as OpenAI / Anthropic etc. Where the real value in applications is smarter prompting techniques and RAG. There is a lot of room at the bottom in doing "dumb" things and simply feeding models with the right context to deliver customer value.

That's a pretty odd stance. I've finetuned llama/mistral models that greatly outperform GPT4 with just a prompt. You have to know when to RAG, finetune, or RAG+finetune.

    greatly outperform GPT4 *for* just a prompt
your overfitting to training data convinces no-one that you created a "better GPT4"

Re: GPT-4, without specialized training, beat a GPT-3.5 class model that cost $10M

#102

Post by rmbyrro »

jebarker wrote: post #16
CharlesW wrote: post #11

Earlier quoted context omitted.

> To me GPT-4 is not a foundation model… It is. https://www.adalovelaceinstitute.org/resource/foundation-mod...

What I was meaning was that ChatGPT is not a foundation model since it's been fine-tuned. Although the definition in the link is sufficiently broad you could choose to include it. I can't tell from the OpenAI docs whether it's possible to access GPT-4 without the ChatGPT fine-tuning. If so, that'd make this result more meaningful. Otherwise, I just don't think you can draw any great conclusions from this.

GPT is general purpose, it's not fine tuned for specific topics. A fine tuned model is tuned to a specific subject.

Re: GPT-4, without specialized training, beat a GPT-3.5 class model that cost $10M

#103

Post by paulmd »

TMWNN wrote: post #94
krasin wrote: post #78

Finetuning LLMs is currently the most promising way for next-gen robotics. One of such works (PaLM-e) among other things measured the impact of finetuning on general purpose tasks: https://twitter.com/DannyDriess/status/1632904698108821505 In short, an 8B model could degrade almost 10x after being finetuned on robotics tasks, while 500B model experiences a very minor degradation (~4%) and there's a hope that with a l…

>there's a hope that with a larger-sized model, it could become either zero or even negative (adding more experience improves general purpose reasoning). The best single work of fiction ever created about LLMs' capabilities (and, perhaps, dangers) is Colossus by Jones. Although I think the film is even better than the book, only the latter mentions how, despite being created specifically for US national defense, Colo…

What good is a revolution without dancing?

Re: GPT-4, without specialized training, beat a GPT-3.5 class model that cost $10M

#104

Post by pbronez »

zapperdulchen wrote: post #40
minimaxir wrote: post #17

Extremely hot LLM take: You will often get better results with few-shot prompting (with good examples) on a modern LLM than with a finetuned LLM. Finetuning was the best option for weaker LLMs with lower context windows (e.g. the original GPT-3): both problems have been solved nowadays. The cost economics are much better with few-shot prompting to modern LLMs too: input tokens are super cheap (especially with the rec…

Seems like the bitter lesson is still right: http://www.incompleteideas.net/IncIdeas/BitterLesson.html

I can’t access the article there… SSL error and then timeout. Here’s a link to the most recent WayBackMachine snapshot:

https://web.archive.org/web/20240321091803/https://www.incom...

Re: GPT-4, without specialized training, beat a GPT-3.5 class model that cost $10M

#105

Post by simonw »

danielmarkbruce wrote: post #88
simonw wrote: post #58

Earlier quoted context omitted.

Citation needed on "trivial to fine tune".

There is no citation needed. It is indeed trivial to fine-tune. Doing a good job is another matter, but the claim is correct. Google around and find a blog post showing how. The claim that RAG is dead is obviously wrong.

For "citation needed", read "please link me to a blog post showing how, don't just tell me to Google for one".

The internet is full of blog posts about this. That doesn't mean they're actually good - I'd love to be pointed at one that has proven itself useful for someone (and definitely isn't just LLM blog-spam).

I don't care if it's trivial to fine-tune and get crap results - I care about fine-tuning where the result was worth the effort.

For the record, my favourite guide to fine-tuning is the section of this Jeremy Howard video that shows how to train a text-to-SQL model: https://www.youtube.com/watch?v=jkrNMKz9pWU&t=4850s

Re: GPT-4, without specialized training, beat a GPT-3.5 class model that cost $10M

#106

Post by bernawil »

potatoman22 wrote: post #32
hulium wrote: post #26

There is also the open source FinGPT, that is claimed to beat GPT4 in some benchmarks at a fine tuning cost of $17.25. https://github.com/AI4Finance-Foundation/FinGPT

One major advantage of FinGPT or Bloomberg's LLM is that the embeddings produced by the model can be used for downstream prediction tasks. GPT-4 does not expose its embeddings so it cannot be used for this.

sorry, noob here trying to make sense of this: you mean you can extract embeddings from the model file or that the embeddings are available in the repo and you can just use those files?

Re: GPT-4, without specialized training, beat a GPT-3.5 class model that cost $10M

#107

Post by redox99 »

kossTKR wrote: post #100
redox99 wrote: post #87

Earlier quoted context omitted.

That's a pretty odd stance. I've finetuned llama/mistral models that greatly outperform GPT4 with just a prompt. You have to know when to RAG, finetune, or RAG+finetune.

How narrow is the dataset to be outperforming greatly? Just curious about what the usecase is for a 7b model in a business context - ie. what does it do?

Code assistant for a niche programming language that GPT4 knows very little about and barely gets a hello world right.

Re: GPT-4, without specialized training, beat a GPT-3.5 class model that cost $10M

#108

Post by danielmarkbruce »

simonw wrote: post #105
danielmarkbruce wrote: post #88

Earlier quoted context omitted.

There is no citation needed. It is indeed trivial to fine-tune. Doing a good job is another matter, but the claim is correct. Google around and find a blog post showing how. The claim that RAG is dead is obviously wrong.

For "citation needed", read "please link me to a blog post showing how, don't just tell me to Google for one". The internet is full of blog posts about this. That doesn't mean they're actually good - I'd love to be pointed at one that has proven itself useful for someone (and definitely isn't just LLM blog-spam). I don't care if it's trivial to fine-tune and get crap results - I care about fine-tuning where the resul…

It's an internet forum, not an academic journal. Water tight arguments are not needed. If one wants to call bs, they can just do it, no need to dance around the topic by asking for a citation.

Re: GPT-4, without specialized training, beat a GPT-3.5 class model that cost $10M

#109

Post by redox99 »

singularity2001 wrote: post #101
redox99 wrote: post #87

Earlier quoted context omitted.

That's a pretty odd stance. I've finetuned llama/mistral models that greatly outperform GPT4 with just a prompt. You have to know when to RAG, finetune, or RAG+finetune.

greatly outperform GPT4 *for* just a prompt your overfitting to training data convinces no-one that you created a "better GPT4"

Do you always assume other people are incompetent? That's not very nice of you.

I mostly work on AI, so I know if I'm overfitting or not. It performs provably better in it's domain (a niche programming language). GPT4 can barely write a hello world for it.

I'm not creating a "better GPT4" general chatbot. I'm finetuning for a specific task.

Re: GPT-4, without specialized training, beat a GPT-3.5 class model that cost $10M

#110

Post by [deleted] »

xanderlewis wrote: post #91
jorvi wrote: post #66

Earlier quoted context omitted.

Compute is.. I don’t know the exact English grammatical term but it’s like water. Computation is not. “I have 1000 flops of compute” - works. “I have 1000 flops of computation” - doesn’t work. “That compute failed” - doesn’t work. “That computation failed” - works. They’re different.

As far as I know, it’s usually called an uncountable noun . …but ‘computation’ is also uncountable, and your second sentence seems to be perfectly fine to me. Your examples do not constitute an argument. You haven’t articulated the (purported) difference between the two words; you’ve just decided arbitrarily that some sentences don’t work, and not elaborated or explained at all. I can make up words too, and provide e…

[deleted]
Post reply on HN