Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GPT-4, without specialized training, beat a GPT-3.5 class model that cost $10M

threads.net

Post reply on HN
Print view View original

31–40 of 134 posts

Re: GPT-4, without specialized training, beat a GPT-3.5 class model that cost $10M

#31

Post by netdur »

viksit wrote: post #23
minimaxir wrote: post #17

Extremely hot LLM take: You will often get better results with few-shot prompting (with good examples) on a modern LLM than with a finetuned LLM. Finetuning was the best option for weaker LLMs with lower context windows (e.g. the original GPT-3): both problems have been solved nowadays. The cost economics are much better with few-shot prompting to modern LLMs too: input tokens are super cheap (especially with the rec…

can you give a few pointers on articles or examples of this?

Use Gemini 1.5 Pro, which has 1.5 million tokens. Prompt it with a logical question and observe it struggling to answer. Then, upload a book on logical thinking in PDF format and ask the same question again. Notice how it can now answer the question effectively.

Re: GPT-4, without specialized training, beat a GPT-3.5 class model that cost $10M

#32

Post by potatoman22 »

hulium wrote: post #26

There is also the open source FinGPT, that is claimed to beat GPT4 in some benchmarks at a fine tuning cost of $17.25. https://github.com/AI4Finance-Foundation/FinGPT

One major advantage of FinGPT or Bloomberg's LLM is that the embeddings produced by the model can be used for downstream prediction tasks. GPT-4 does not expose its embeddings so it cannot be used for this.

Re: GPT-4, without specialized training, beat a GPT-3.5 class model that cost $10M

#33

Post by viksit »

minimaxir wrote: post #30
viksit wrote: post #23

Earlier quoted context omitted.

can you give a few pointers on articles or examples of this?

A low-tech example to create a good blog post title for submission to Hacker News would be a system prompt like: You are an expert copywriter. Write five distinct blog post titles optimized for high clickthrough for Hacker News for the article the user provides. Your response must follow the style of these titles: - The ü/ü Conundrum - Why isn't preprint review being adopted? - Majority of web apps could just run on…

interesting thank you.

intuitively, prompting like this to get an answer seems basically like the first part of a fine tuning process (more exemplars).

what is your thought here behind why reinforcing good output via a loss optimization is worse than the one shot example? does the model start to over fit at some point towards some local minima? and this is avoided in this scenario?

Re: GPT-4, without specialized training, beat a GPT-3.5 class model that cost $10M

#34

Post by doctorpangloss »

jebarker wrote: post #16
CharlesW wrote: post #11

Earlier quoted context omitted.

> To me GPT-4 is not a foundation model… It is. https://www.adalovelaceinstitute.org/resource/foundation-mod...

What I was meaning was that ChatGPT is not a foundation model since it's been fine-tuned. Although the definition in the link is sufficiently broad you could choose to include it. I can't tell from the OpenAI docs whether it's possible to access GPT-4 without the ChatGPT fine-tuning. If so, that'd make this result more meaningful. Otherwise, I just don't think you can draw any great conclusions from this.

The instruction fine tuning is what manifests knowledge and reasoning.

Re: GPT-4, without specialized training, beat a GPT-3.5 class model that cost $10M

#35

Post by thorum »

minimaxir wrote: post #17

Extremely hot LLM take: You will often get better results with few-shot prompting (with good examples) on a modern LLM than with a finetuned LLM. Finetuning was the best option for weaker LLMs with lower context windows (e.g. the original GPT-3): both problems have been solved nowadays. The cost economics are much better with few-shot prompting to modern LLMs too: input tokens are super cheap (especially with the rec…

That might be true for finetuning ChatGPT 3.5, but if you can finetune a small model (7B or less) to perform on par with GPT-4, while being faster and private, that’s a different story.

Re: GPT-4, without specialized training, beat a GPT-3.5 class model that cost $10M

#36

Post by yieldcrv »

MuffinFlavored wrote: post #4

Does anything currently beat GPT-4? I saw some comments here say to check out Claude. From what I can tell, Claude hasn't figure out yet how to do the whole "generate Python code and run it in a Juptyer notebook" for math yet.

The benchmark is Sora or whatever Open AI is working on right now or next, not trying to beat the model released a year ago and still failing

so when looking at it that way, the real question is what do you need? all I need is Mixtral 7x8B Q5 in an 8,000 token context window, at the moment

I think there are plenty of other people that can design their applications and problems around lower fidelity tools, or just pursue something else

Re: GPT-4, without specialized training, beat a GPT-3.5 class model that cost $10M

#37

Post by mistrial9 »

ShamelessC wrote: post #21
rafaelero wrote: post #19

Earlier quoted context omitted.

A lot of organizations still think they should have their own [finetuned] model to provide a custom experience to their users, so that may come as a surprise for them.

Scaling laws basically guarantee that a sufficiently larger general model will usually beat a smaller specialist model. The misunderstanding is perhaps acceptable but the headline here is essentially restating a well known property of deep learning.

contrarian view - how these models actually operate at runtime is not understood.. the formal research papers repeat that over and over again. Therefore, there will be new twists and turns as these models evolve. With current technology stacks, the "bitter lesson" is looking good, yes. Will it always be so? no way to know it.

Re: GPT-4, without specialized training, beat a GPT-3.5 class model that cost $10M

#38

Post by [deleted] »

ldjkfkdsjnv wrote: post #28
minimaxir wrote: post #24

Earlier quoted context omitted.

> Also, most LLM code being written right now (prompts, RAG, etc) will be obsolete once the next model comes out. Not true. Most prompt techniques that work on current modern LLM models will work on different or future models, although it will require a QA pass for any regressions.

Yeah they will work, but they will also be unnecessary. You will also bake all sorts of logic into your application that will be solved natively on the stronger model. If you believe in the scaling theory, then writing LLM applications is non sensical.

[deleted]

Re: GPT-4, without specialized training, beat a GPT-3.5 class model that cost $10M

#39

Post by mlyle »

ldjkfkdsjnv wrote: post #28
minimaxir wrote: post #24

Earlier quoted context omitted.

> Also, most LLM code being written right now (prompts, RAG, etc) will be obsolete once the next model comes out. Not true. Most prompt techniques that work on current modern LLM models will work on different or future models, although it will require a QA pass for any regressions.

Yeah they will work, but they will also be unnecessary. You will also bake all sorts of logic into your application that will be solved natively on the stronger model. If you believe in the scaling theory, then writing LLM applications is non sensical.

> Yeah they will work, but they will also be unnecessary. You will also bake all sorts of logic into your application that will be solved natively on the stronger model.

I think that a whole lot of what I do in prompt engineering is what's necessary to fully specify the output that I want.

A newer model may be less finicky, so I have a higher chance of getting it to work on the first try (and it's more reliable afterwards), but it's hard for me to imagine it needing a whole lot less prompt.

Re: GPT-4, without specialized training, beat a GPT-3.5 class model that cost $10M

#40

Post by zapperdulchen »

minimaxir wrote: post #17

Extremely hot LLM take: You will often get better results with few-shot prompting (with good examples) on a modern LLM than with a finetuned LLM. Finetuning was the best option for weaker LLMs with lower context windows (e.g. the original GPT-3): both problems have been solved nowadays. The cost economics are much better with few-shot prompting to modern LLMs too: input tokens are super cheap (especially with the rec…

Seems like the bitter lesson is still right: http://www.incompleteideas.net/IncIdeas/BitterLesson.html
Post reply on HN