Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GPT-4, without specialized training, beat a GPT-3.5 class model that cost $10M

threads.net

Post reply on HN
Print view View original

11–20 of 134 posts

Re: GPT-4, without specialized training, beat a GPT-3.5 class model that cost $10M

#11

Post by CharlesW »

jebarker wrote: post #9
CharlesW wrote: post #7

Earlier quoted context omitted.

Meaning, they used the same generalized foundation model that all of us have access to, with no special fine-tuning, no retrieval-augmented generation, etc.

I don't understand your point. To me GPT-4 is not a foundation model, it's been highly tuned for the chat task. Nobody outside of OpenAI knows what that fine-tuning really involved. So it's impossible to say how much finance specific data it was trained on (in pre-training or fine-tuning) or whether finance specific tasks were involved in fine-tuning.

> To me GPT-4 is not a foundation model…

It is. https://www.adalovelaceinstitute.org/resource/foundation-mod...

Re: GPT-4, without specialized training, beat a GPT-3.5 class model that cost $10M

#12

Post by MuffinFlavored »

monsieurbanana wrote: post #8
MuffinFlavored wrote: post #4

Does anything currently beat GPT-4? I saw some comments here say to check out Claude. From what I can tell, Claude hasn't figure out yet how to do the whole "generate Python code and run it in a Juptyer notebook" for math yet.

Isn't that something you get from the infrastructure surrounding the llm? I thought the "running code" feature didn't need specific support from the llm, besides being able to output conforming json or code when asked to.

The LLM (Claude) currently doesn't know to not hallucinate numbers and instead write code + run it (something ChatGPT used to do but they fixed it)

Re: GPT-4, without specialized training, beat a GPT-3.5 class model that cost $10M

#13

Post by [deleted] »

CharlesW wrote: post #11
jebarker wrote: post #9

Earlier quoted context omitted.

I don't understand your point. To me GPT-4 is not a foundation model, it's been highly tuned for the chat task. Nobody outside of OpenAI knows what that fine-tuning really involved. So it's impossible to say how much finance specific data it was trained on (in pre-training or fine-tuning) or whether finance specific tasks were involved in fine-tuning.

> To me GPT-4 is not a foundation model… It is. https://www.adalovelaceinstitute.org/resource/foundation-mod...

[deleted]

Re: GPT-4, without specialized training, beat a GPT-3.5 class model that cost $10M

#14

Post by geor9e »

MuffinFlavored wrote: post #4

Does anything currently beat GPT-4? I saw some comments here say to check out Claude. From what I can tell, Claude hasn't figure out yet how to do the whole "generate Python code and run it in a Juptyer notebook" for math yet.

I don't know what system and user prompt you are testing with, but as one anecdote, Claude 3 Opus (and only Opus) consistently gives me better coding answers than GPT-4. Maybe it's the type of stuff I am doing or how I phrase things, who knows. I was using GPT-4 since the day it came out but haven't felt like going back so far.

Re: GPT-4, without specialized training, beat a GPT-3.5 class model that cost $10M

#16

Post by jebarker »

CharlesW wrote: post #11
jebarker wrote: post #9

Earlier quoted context omitted.

I don't understand your point. To me GPT-4 is not a foundation model, it's been highly tuned for the chat task. Nobody outside of OpenAI knows what that fine-tuning really involved. So it's impossible to say how much finance specific data it was trained on (in pre-training or fine-tuning) or whether finance specific tasks were involved in fine-tuning.

> To me GPT-4 is not a foundation model… It is. https://www.adalovelaceinstitute.org/resource/foundation-mod...

What I was meaning was that ChatGPT is not a foundation model since it's been fine-tuned. Although the definition in the link is sufficiently broad you could choose to include it.

I can't tell from the OpenAI docs whether it's possible to access GPT-4 without the ChatGPT fine-tuning. If so, that'd make this result more meaningful. Otherwise, I just don't think you can draw any great conclusions from this.

Re: GPT-4, without specialized training, beat a GPT-3.5 class model that cost $10M

#17

Post by minimaxir »

Extremely hot LLM take: You will often get better results with few-shot prompting (with good examples) on a modern LLM than with a finetuned LLM.

Finetuning was the best option for weaker LLMs with lower context windows (e.g. the original GPT-3): both problems have been solved nowadays.

The cost economics are much better with few-shot prompting to modern LLMs too: input tokens are super cheap (especially with the recently-released Claude Haiku), so giving a lot of examples per call will still end up cheaper than finetuning.

Meanwhile, a finetuned ChatGPT costs 4-6x of normal ChatGPT usage.

Re: GPT-4, without specialized training, beat a GPT-3.5 class model that cost $10M

#18

Post by avree »

MuffinFlavored wrote: post #12
monsieurbanana wrote: post #8

Earlier quoted context omitted.

Isn't that something you get from the infrastructure surrounding the llm? I thought the "running code" feature didn't need specific support from the llm, besides being able to output conforming json or code when asked to.

The LLM (Claude) currently doesn't know to not hallucinate numbers and instead write code + run it (something ChatGPT used to do but they fixed it)

Doesn't seem like you are very informed on how LLMs work, but just so you know, there are many different versions of Claude, just like how ChatGPT can use different versions of GPT.

Re: GPT-4, without specialized training, beat a GPT-3.5 class model that cost $10M

#19

Post by rafaelero »

__loam wrote: post #15

GPT-4 cost like $100m so I don't think this is surprising?

A lot of organizations still think they should have their own [finetuned] model to provide a custom experience to their users, so that may come as a surprise for them.

Re: GPT-4, without specialized training, beat a GPT-3.5 class model that cost $10M

#20

Post by jxdxbx »

MuffinFlavored wrote: post #4

Does anything currently beat GPT-4? I saw some comments here say to check out Claude. From what I can tell, Claude hasn't figure out yet how to do the whole "generate Python code and run it in a Juptyer notebook" for math yet.

Claude is better than GPT 4 for my uses, and was able to help me do some simple coding things that GPT 4 could not. It’s worth trying at least.
Post reply on HN