Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Benchmarking GPT-4 Turbo – A Cautionary Tale

blog.mentat.ai

Post reply on HN
Print view View original

51–60 of 118 posts

Re: Benchmarking GPT-4 Turbo – A Cautionary Tale

#51

Post by Terretta »

boxed wrote: post #14
famouswaffles wrote: post #7

Earlier quoted context omitted.

"They memorized all the problems" is not what was found here and still a wrong overcorrection.

If you are shown only the title of a coding problem and the site name where it's from, and you manage to solve it you are showing that you either cheated or knew the answer.

On the contrary, it could mean you were, to some percentage of success, able to guess what problem is, and then, to some multiplier percentage of success, solve it.

The key is, can you guess the problem from the title and the function name? I'd argue, sure, at least half the time?, why not...

Re: Benchmarking GPT-4 Turbo – A Cautionary Tale

#52

Post by raincole »

boxed wrote: post #48
raincole wrote: post #38

Earlier quoted context omitted.

Me: What's 6x6? You: 36 Me: You cheated! You just cited the answer you memorized! You should have started from addition. You: ...okay? 6+6=12, 12+6=18, 18+... Me: You cheated again! You just have 6+6=12 memorized! You should make the rule of addition out of Peano axioms. You: ...you're being annoying, but okay? First axiom, we define 0 as... Me: You cheated again ! You memorized Peano Axioms! Jesus Christ, is there a…

But in this case it's not like that at all. They only saw the NAME of the problem. Like if I said "Page 23 of Mathbook Y, problem number 3". Which happens to be 6x6.

Me: I was being in such a blah blah situation... does the article 3 of Digital Government Act applies here?

My lawyer: Hmm the article 3 says--

Me: I knew it! Lawyers are not intelligent...

Re: Benchmarking GPT-4 Turbo – A Cautionary Tale

#53

Post by jphoward »

The problem is the discussed results are comparing proportions of a relatively small number - 67 questions. If you model this as a binomial distribution, then 62/67 which GPT4-turbo got gives a 95% confidence interval of the 'true' performance of 83.4% to 97.5%, ie it comfortably includes the proportion that GPT4 achieved (64/67=95.5%).

I think the evidence from these tests are not strong enough to draw conclusions from.

Re: Benchmarking GPT-4 Turbo – A Cautionary Tale

#54

Post by furyofantares »

boxed wrote: post #48
raincole wrote: post #38

Earlier quoted context omitted.

Me: What's 6x6? You: 36 Me: You cheated! You just cited the answer you memorized! You should have started from addition. You: ...okay? 6+6=12, 12+6=18, 18+... Me: You cheated again! You just have 6+6=12 memorized! You should make the rule of addition out of Peano axioms. You: ...you're being annoying, but okay? First axiom, we define 0 as... Me: You cheated again ! You memorized Peano Axioms! Jesus Christ, is there a…

But in this case it's not like that at all. They only saw the NAME of the problem. Like if I said "Page 23 of Mathbook Y, problem number 3". Which happens to be 6x6.

It said they gave the exercise name, which doesn't sound like just the exercise number but probably mildly descriptive -- and they also gave it function stubs.

Re: Benchmarking GPT-4 Turbo – A Cautionary Tale

#55

Post by iJohnDoe »

A little bit off topic question. When people are talking about costs with GPT, like the following link. Does the cost concern only apply to the API? If you’re using the WebUI and have a Plus account, is it always just the flat $20 amount?

https://news.ycombinator.com/item?id=38193978

Re: Benchmarking GPT-4 Turbo – A Cautionary Tale

#56

Post by geraltofrivia »

In my day job we use GPT4 quite a bit and we shifted to GPT4 Turbo today. We got a 2-5% performance increase, and quite a bit of speed increase as well.

Not to say that the parent post is incorrect, of course. Only that its not as cut and dry as a "GPT4 Turbo is distilled (read: watered down) GPT4".

Re: Benchmarking GPT-4 Turbo – A Cautionary Tale

#58

Post by Satam »

Very interesting and basically confirms that GPT-4 turbo is a faster but dumber model. When a task doesn't rely on memorization of the training set, it reasons similarly well to GPT-4. Where memorization is helpful, it performs worse (due to quantization-induced "memory loss").

This also makes me look at GPT-4 as a "weak reasoner with a lot of knowledge". That really aligns with my experience where it is immensely helpful and has a superhuman knowledge base but still needs handholding to solve real problems.

Re: Benchmarking GPT-4 Turbo – A Cautionary Tale

#59

Post by rocgf »

ChatGTP wrote: post #57
stoicfungi wrote: post #15

Earlier quoted context omitted.

All models are, including humain brain.

The human brain is a model?

It models the world around it, so it's fairly similar to what GPT does, especially with the newly-added image capabilities and stuff.

Re: Benchmarking GPT-4 Turbo – A Cautionary Tale

#60

Post by stnmtn »

boxed wrote: post #48
raincole wrote: post #38

Earlier quoted context omitted.

Me: What's 6x6? You: 36 Me: You cheated! You just cited the answer you memorized! You should have started from addition. You: ...okay? 6+6=12, 12+6=18, 18+... Me: You cheated again! You just have 6+6=12 memorized! You should make the rule of addition out of Peano axioms. You: ...you're being annoying, but okay? First axiom, we define 0 as... Me: You cheated again ! You memorized Peano Axioms! Jesus Christ, is there a…

But in this case it's not like that at all. They only saw the NAME of the problem. Like if I said "Page 23 of Mathbook Y, problem number 3". Which happens to be 6x6.

If I gave you a programming problem and all I told you was that the problem name was Traveling Salesman, you might be able to solve it based on that.

If not that, then if I just said "fizzbuzz" to you, I'm sure you would be able to give the solution without me needing to say any other descriptions of the problem

Post reply on HN