Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GPT-4, without specialized training, beat a GPT-3.5 class model that cost $10M

threads.net

Post reply on HN
Print view View original

81–90 of 134 posts

Re: GPT-4, without specialized training, beat a GPT-3.5 class model that cost $10M

#81

Post by fragmede »

simonw wrote: post #53
MuffinFlavored wrote: post #12

Earlier quoted context omitted.

The LLM (Claude) currently doesn't know to not hallucinate numbers and instead write code + run it (something ChatGPT used to do but they fixed it)

That's because the Claude web UI doesn't yet have the equivalent of the ChatGPT Code Interpreter tool (though they say they're working on it). That's not about the quality of the Claude 3 Opus model, which is the model which people think compares to or beats GPT-4. It's about the tooling that has been built for ChatGPT.

Code interpreter is pretty neat, because you can tell ChatGPT to write some code and to make sure the code works, and then it'll write you some bad code, realize it's bad, and then iterate on it until it gets to a place that it's happy with. (Maybe I should say passes its test rather than anthropomorphize ChatGPT as being "happy".)

Re: GPT-4, without specialized training, beat a GPT-3.5 class model that cost $10M

#82

Post by greenavocado »

Xenoamorphous wrote: post #56
jnwatson wrote: post #49

Earlier quoted context omitted.

It is trivial to fine tune these days. RAG is already irrelevant with large context windows.

> RAG is already irrelevant with large context windows Just last Friday I took the contents of the 2024 folder of one of the teams at the company I work for, for which we use RAG at the moment. I dumped the text index, concatenated it and used Google’s API to return the token count, to see if it would fit in Gemini’s 1M context window; turned out it was 5.7M tokens. And that’s less than 3 months worth of documents fo…

Did I read this correctly? You uploaded millions of words of your company's internal communications to Google?

Re: GPT-4, without specialized training, beat a GPT-3.5 class model that cost $10M

#83

Post by dragonwriter »

MuffinFlavored wrote: post #4

Does anything currently beat GPT-4? I saw some comments here say to check out Claude. From what I can tell, Claude hasn't figure out yet how to do the whole "generate Python code and run it in a Juptyer notebook" for math yet.

> Claude hasn't figure out yet how to do the whole "generate Python code and run it in a Juptyer notebook" for math yet.

GPT-4 didn't figure that out, either; that’s just tooling built around the model, not something the model “figures out”.

Re: GPT-4, without specialized training, beat a GPT-3.5 class model that cost $10M

#84

Post by fragmede »

rubymamis wrote: post #10
MuffinFlavored wrote: post #4

Does anything currently beat GPT-4? I saw some comments here say to check out Claude. From what I can tell, Claude hasn't figure out yet how to do the whole "generate Python code and run it in a Juptyer notebook" for math yet.

A programming task where Mistral-large beats both GPT-4 and Claude Opus: https://sdk.vercel.ai/s/Thi7RDx9e8VOZo1Ee6We5 (only Mistral got the current syntax) Although based on other tasks, overall, GPT-4 seems to be the best, but by a very small margin, so I cancelled my subscription. Although the native mobile app is really great.

Is there a way to use Mistral-large with TTS and STT engines so you can converse with it like you can ChatGPT in the mobile app? it's really great on long drives for learning/talking about stuff, like a customized personal podcast.

Re: GPT-4, without specialized training, beat a GPT-3.5 class model that cost $10M

#85

Post by tomrod »

zapperdulchen wrote: post #40
minimaxir wrote: post #17

Extremely hot LLM take: You will often get better results with few-shot prompting (with good examples) on a modern LLM than with a finetuned LLM. Finetuning was the best option for weaker LLMs with lower context windows (e.g. the original GPT-3): both problems have been solved nowadays. The cost economics are much better with few-shot prompting to modern LLMs too: input tokens are super cheap (especially with the rec…

Seems like the bitter lesson is still right: http://www.incompleteideas.net/IncIdeas/BitterLesson.html

This is an earth-shattering read.

Re: GPT-4, without specialized training, beat a GPT-3.5 class model that cost $10M

#86

Post by atleastoptimal »

Yeah, the equivalent is: would it be better for a quant firm to spend 200 thousand dollars giving a first-class specialist education to a guy with an IQ of 95, or just hiring a guy with an IQ of 150 straight out of college.

Re: GPT-4, without specialized training, beat a GPT-3.5 class model that cost $10M

#87

Post by redox99 »

LASR wrote: post #42

I lead AI teams at my company. I've advised leadership against any kind of training / fine-tuning anything. We're not in the business of training models. We will never be as good as OpenAI / Anthropic etc. Where the real value in applications is smarter prompting techniques and RAG. There is a lot of room at the bottom in doing "dumb" things and simply feeding models with the right context to deliver customer value.

That's a pretty odd stance. I've finetuned llama/mistral models that greatly outperform GPT4 with just a prompt.

You have to know when to RAG, finetune, or RAG+finetune.

Re: GPT-4, without specialized training, beat a GPT-3.5 class model that cost $10M

#88

Post by danielmarkbruce »

simonw wrote: post #58
jnwatson wrote: post #49

Earlier quoted context omitted.

It is trivial to fine tune these days. RAG is already irrelevant with large context windows.

Citation needed on "trivial to fine tune".

There is no citation needed. It is indeed trivial to fine-tune. Doing a good job is another matter, but the claim is correct. Google around and find a blog post showing how.

The claim that RAG is dead is obviously wrong.

Re: GPT-4, without specialized training, beat a GPT-3.5 class model that cost $10M

#89

Post by kcorbitt »

IMO it's possible to over-generalize from this datapoint (lol). While it's true that creating a general "finance" model that's stronger than GPT-4 is hard, training a task-specific model is much easier. Eg. "a model that's better than GPT-4 at answering finance-related questions": very hard. "A model that's better than GPT-4 at extracting forward-looking financial projections in a standard format": very easy.

And in practice, most tasks people are using GPT-4 for in production are more like the latter than the former.

(Disclaimer: building https://openpipe.ai, which makes it super easy to productize this workflow).

Post reply on HN