Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GPT-4, without specialized training, beat a GPT-3.5 class model that cost $10M

threads.net

Post reply on HN
Print view View original

51–60 of 134 posts

Re: GPT-4, without specialized training, beat a GPT-3.5 class model that cost $10M

#51

Post by Solvency »

zapperdulchen wrote: post #40
minimaxir wrote: post #17

Extremely hot LLM take: You will often get better results with few-shot prompting (with good examples) on a modern LLM than with a finetuned LLM. Finetuning was the best option for weaker LLMs with lower context windows (e.g. the original GPT-3): both problems have been solved nowadays. The cost economics are much better with few-shot prompting to modern LLMs too: input tokens are super cheap (especially with the rec…

Seems like the bitter lesson is still right: http://www.incompleteideas.net/IncIdeas/BitterLesson.html

Whoa this guy says "computation" and not grammatically bastardized techbrospeak "compute" like some neckbeard equivalent of a caveman!

For that alone I commend him.

Re: GPT-4, without specialized training, beat a GPT-3.5 class model that cost $10M

#52

Post by Xenoamorphous »

ldjkfkdsjnv wrote: post #28
minimaxir wrote: post #24

Earlier quoted context omitted.

> Also, most LLM code being written right now (prompts, RAG, etc) will be obsolete once the next model comes out. Not true. Most prompt techniques that work on current modern LLM models will work on different or future models, although it will require a QA pass for any regressions.

Yeah they will work, but they will also be unnecessary. You will also bake all sorts of logic into your application that will be solved natively on the stronger model. If you believe in the scaling theory, then writing LLM applications is non sensical.

How will a more powerful model be a substitute for RAG, which is usually used with private data that won’t be present in any training dataset?

Re: GPT-4, without specialized training, beat a GPT-3.5 class model that cost $10M

#53

Post by simonw »

MuffinFlavored wrote: post #12
monsieurbanana wrote: post #8

Earlier quoted context omitted.

Isn't that something you get from the infrastructure surrounding the llm? I thought the "running code" feature didn't need specific support from the llm, besides being able to output conforming json or code when asked to.

The LLM (Claude) currently doesn't know to not hallucinate numbers and instead write code + run it (something ChatGPT used to do but they fixed it)

That's because the Claude web UI doesn't yet have the equivalent of the ChatGPT Code Interpreter tool (though they say they're working on it). That's not about the quality of the Claude 3 Opus model, which is the model which people think compares to or beats GPT-4. It's about the tooling that has been built for ChatGPT.

Re: GPT-4, without specialized training, beat a GPT-3.5 class model that cost $10M

#54

Post by rkagerer »

ldjkfkdsjnv wrote: post #22

Fine tuning will disappear, no reason to invest so heavily in it. Also, most LLM code being written right now (prompts, RAG, etc) will be obsolete once the next model comes out. Anyone starting an LLM application startup is arguably wasting their time, wait until the next iteration is out. Then you will know whats possible.

"Don't buy a computer today, because the faster one is coming out tomorrow"

Re: GPT-4, without specialized training, beat a GPT-3.5 class model that cost $10M

#55

Post by simonw »

thorum wrote: post #43
MuffinFlavored wrote: post #4

Does anything currently beat GPT-4? I saw some comments here say to check out Claude. From what I can tell, Claude hasn't figure out yet how to do the whole "generate Python code and run it in a Juptyer notebook" for math yet.

I don’t know what the people who say Claude 3 is better than GPT-4 are using it for. It’s been consistently worse for everything I’ve thrown at it. Debugging a Python function this morning. Claude 3 Opus failed completely. GPT-4 found the bug, as well as two others I hadn’t even been looking for.

I've had the opposite experience: coding prompts that GPT-4 makes mistakes on Claude 3 Opus gets right the first time.

As always, your results will vary based on your personal prompting style. My style apparently works great with Opus.

Here's one example: GPT-4 gave me code that was missing some async/await keywords: https://chat.openai.com/share/117fb1ad-6361-41e2-be59-110f32...

Claude 3 Opus with the same prompt got it right the first time: https://gist.github.com/simonw/2002e2b56a97053bd9302a34e0b83...

Re: GPT-4, without specialized training, beat a GPT-3.5 class model that cost $10M

#56

Post by Xenoamorphous »

jnwatson wrote: post #49
LASR wrote: post #42

I lead AI teams at my company. I've advised leadership against any kind of training / fine-tuning anything. We're not in the business of training models. We will never be as good as OpenAI / Anthropic etc. Where the real value in applications is smarter prompting techniques and RAG. There is a lot of room at the bottom in doing "dumb" things and simply feeding models with the right context to deliver customer value.

It is trivial to fine tune these days. RAG is already irrelevant with large context windows.

> RAG is already irrelevant with large context windows

Just last Friday I took the contents of the 2024 folder of one of the teams at the company I work for, for which we use RAG at the moment. I dumped the text index, concatenated it and used Google’s API to return the token count, to see if it would fit in Gemini’s 1M context window; turned out it was 5.7M tokens. And that’s less than 3 months worth of documents for that team.

So yeah RAG is not dead yet, although I do question its usefulness, but that’s a separate topic.

Re: GPT-4, without specialized training, beat a GPT-3.5 class model that cost $10M

#57

Post by ldjkfkdsjnv »

Xenoamorphous wrote: post #52
ldjkfkdsjnv wrote: post #28

Earlier quoted context omitted.

Yeah they will work, but they will also be unnecessary. You will also bake all sorts of logic into your application that will be solved natively on the stronger model. If you believe in the scaling theory, then writing LLM applications is non sensical.

How will a more powerful model be a substitute for RAG, which is usually used with private data that won’t be present in any training dataset?

I just think that the capability of the model could radically change, such that however you structured your RAG pipeline, might need to be rewritten. More general problems could be solved by the model, that you were solving with some complicated contraption of prompts.

Re: GPT-4, without specialized training, beat a GPT-3.5 class model that cost $10M

#58

Post by simonw »

jnwatson wrote: post #49
LASR wrote: post #42

I lead AI teams at my company. I've advised leadership against any kind of training / fine-tuning anything. We're not in the business of training models. We will never be as good as OpenAI / Anthropic etc. Where the real value in applications is smarter prompting techniques and RAG. There is a lot of room at the bottom in doing "dumb" things and simply feeding models with the right context to deliver customer value.

It is trivial to fine tune these days. RAG is already irrelevant with large context windows.

Citation needed on "trivial to fine tune".

Re: GPT-4, without specialized training, beat a GPT-3.5 class model that cost $10M

#60

Post by seydor »

LASR wrote: post #42

I lead AI teams at my company. I've advised leadership against any kind of training / fine-tuning anything. We're not in the business of training models. We will never be as good as OpenAI / Anthropic etc. Where the real value in applications is smarter prompting techniques and RAG. There is a lot of room at the bottom in doing "dumb" things and simply feeding models with the right context to deliver customer value.

Your advise is based on what?
Post reply on HN