Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Benchmarking GPT-4 Turbo – A Cautionary Tale

blog.mentat.ai

Post reply on HN
Print view View original

41–50 of 118 posts

Re: Benchmarking GPT-4 Turbo – A Cautionary Tale

#41

Post by m3kw9 »

boxed wrote: post #4

> We designed a test for this theory: we reran the benchmarks without showing the models the instructions to each exercise. Instead, we just told them that they were Exercism exercises, and gave them the exercise names and function stubs. This summarizes all my skepticism agains the AI field. Pretty clear that they aren't solving the problem, they have them memorized.

From a black box point of view and one angle, gpt is a web filter where it will try to find you the exact thing you are looking for but from memory. Vs google you have to distill all the info into what you need

Re: Benchmarking GPT-4 Turbo – A Cautionary Tale

#44

Post by msp26 »

phillipcarter wrote: post #35
msp26 wrote: post #25

I'm interested in more testing on the context side of things. For my NLP pipelines, I batch n-articles together to process (extract fields from) in one prompt (final output is something like this {"1":[{}], "2": [{},{}]...}) in one message. Compute-wise it's inefficient but OpenAI charges by the token so it doesn't matter. It's very reliable on gpt-4 8k. I was also pretty happy with the results on 4-turbo initially b…

Interesting. I was skeptical about some of their claims regarding longer context, since it's been my experience that these models just get lost after enough of it.

Yeah, degraded performance on long contexts has been observed in plenty of other models [https://arxiv.org/abs/2307.03172] so I was cautious too. Unfortunately I don't have access to 4-32k. I would have liked to test that out too.

Re: Benchmarking GPT-4 Turbo – A Cautionary Tale

#45

Post by boxed »

hanselot wrote: post #36
famouswaffles wrote: post #23

Earlier quoted context omitted.

Okay... Funny how forcing it to not CHEAT did not increase apparent ability. It can code and it has memorized some coding questions are not mutually exclusive.

Though this is exactly what happened. The initial test was ran on a model that "Cheated" (aka has memorized the answers). The second test was run on a model that didn't "Cheat" as much, yet still got only 2% less score. So, the question is not resolved really. How much did the first model cheat, and how much did the second? If the second model "cheats" less, then it wins. Also, I don't understand your obsession with…

> Also, I don't understand your obsession with the word cheating.

It's all about the rule set yea. Since the rule set is not defined, technically nothing is cheating. I just interpret the rule set as "can it code?" and for this rule set, it seems to me that it's cheating.

Re: Benchmarking GPT-4 Turbo – A Cautionary Tale

#46

Post by minihat »

GPT-4 Turbo is dramatically worse at one task I often try:

Read the following passage from [new ML article]. Identify their assumptions, and tell me which mathematical operations or procedures they use depend upon these assumptions.

GPT-4: Usually correctly identifies the assumptions, and often quotes the correct mathematics in its reply.

GPT-4 Turbo: Sometimes identifies the assumptions, and is guaranteed to stop trying at that point and then give me a Wikipedia-like summary about the assumptions rather than finish the task. Further prompting will not improve its result.

Re: Benchmarking GPT-4 Turbo – A Cautionary Tale

#47

Post by derwiki »

anotherpaulg wrote: post #12

Aider has had an Exercism benchmarking suite for quite some time. Interestingly, my benchmark results of GPT 4 Turbo show an opposite result: the new gpt-4-1106-preview did significantly better on the first try than the March and June models. https://aider.chat/docs/benchmarks-1106.html Aider benchmarks against the 133 Exercism python exercises, not js exercises that mentat's benchmark uses. So this is not an apples-…

I’ve been using the new model with Aider since it was released, and my anecdata agrees—the “edits applied successfully “ failure rate is much lower than classic gpt4.

Also THANK YOU for Aider! I talk it up to all my programmer friends; it really feels like a glimpse into the future of coding.

Re: Benchmarking GPT-4 Turbo – A Cautionary Tale

#48

Post by boxed »

raincole wrote: post #38
boxed wrote: post #31

Earlier quoted context omitted.

> Okay... Funny how forcing it to not CHEAT did not increase apparent ability. The article did the opposite. It forced the models to cheat to solve the problems. Which it did happily. It should have stated "there is no actual problem to solve here, you must supply a problem for me to solve". > It can code and it has memorized some coding questions are not mutually exclusive This I will give you. Many humans try to ch…

Me: What's 6x6? You: 36 Me: You cheated! You just cited the answer you memorized! You should have started from addition. You: ...okay? 6+6=12, 12+6=18, 18+... Me: You cheated again! You just have 6+6=12 memorized! You should make the rule of addition out of Peano axioms. You: ...you're being annoying, but okay? First axiom, we define 0 as... Me: You cheated again ! You memorized Peano Axioms! Jesus Christ, is there a…

But in this case it's not like that at all. They only saw the NAME of the problem. Like if I said "Page 23 of Mathbook Y, problem number 3". Which happens to be 6x6.

Re: Benchmarking GPT-4 Turbo – A Cautionary Tale

#50

Post by boxed »

hanselot wrote: post #36
famouswaffles wrote: post #23

Earlier quoted context omitted.

Okay... Funny how forcing it to not CHEAT did not increase apparent ability. It can code and it has memorized some coding questions are not mutually exclusive.

Though this is exactly what happened. The initial test was ran on a model that "Cheated" (aka has memorized the answers). The second test was run on a model that didn't "Cheat" as much, yet still got only 2% less score. So, the question is not resolved really. How much did the first model cheat, and how much did the second? If the second model "cheats" less, then it wins. Also, I don't understand your obsession with…

> How much did the first model cheat, and how much did the second? If the second model "cheats" less, then it wins.

They both cheated 100%. Because they both never saw the problem. AT ALL. They just saw the title and the name of the website.

Post reply on HN