[deleted]
Benchmarking GPT-4 Turbo – A Cautionary Tale
11–20 of 118 posts
Re: Benchmarking GPT-4 Turbo – A Cautionary Tale
#12Aider has had an Exercism benchmarking suite for quite some time.
Interestingly, my benchmark results of GPT 4 Turbo show an opposite result: the new gpt-4-1106-preview did significantly better on the first try than the March and June models.
https://aider.chat/docs/benchmarks-1106.html
Aider benchmarks against the 133 Exercism python exercises, not js exercises that mentat's benchmark uses. So this is not an apples-to-apples comparison, but there doesn't seem to be a strong reason to expect qualitatively different results.
I also notice that the instructions prompt that mentat uses seems to be inspired by the aider benchmark? Glad to see others adopting similar benchmarking approaches.
https://github.com/AbanteAI/mentat/blob/main/tests/benchmark...
https://github.com/paul-gauthier/aider/blob/main/benchmark/p...
Edit: Not sure if the mentat authors are in this thread? After looking around a bit, there seems to be a bunch of aider code in your repo. Some attribution would be appreciated. It might even be required under aider's Apache 2.0 license?
Re: Benchmarking GPT-4 Turbo – A Cautionary Tale
#13Aider has had an Exercism benchmarking suite for quite some time. Interestingly, my benchmark results of GPT 4 Turbo show an opposite result: the new gpt-4-1106-preview did significantly better on the first try than the March and June models. https://aider.chat/docs/benchmarks-1106.html Aider benchmarks against the 133 Exercism python exercises, not js exercises that mentat's benchmark uses. So this is not an apples-…
I am also noticing a massive improvement over the old model
Re: Benchmarking GPT-4 Turbo – A Cautionary Tale
#14> We designed a test for this theory: we reran the benchmarks without showing the models the instructions to each exercise. Instead, we just told them that they were Exercism exercises, and gave them the exercise names and function stubs. This summarizes all my skepticism agains the AI field. Pretty clear that they aren't solving the problem, they have them memorized.
"They memorized all the problems" is not what was found here and still a wrong overcorrection.
If you are shown only the title of a coding problem and the site name where it's from, and you manage to solve it you are showing that you either cheated or knew the answer.
Re: Benchmarking GPT-4 Turbo – A Cautionary Tale
#15> We designed a test for this theory: we reran the benchmarks without showing the models the instructions to each exercise. Instead, we just told them that they were Exercism exercises, and gave them the exercise names and function stubs. This summarizes all my skepticism agains the AI field. Pretty clear that they aren't solving the problem, they have them memorized.
LLMs are lossy compression
All models are, including humain brain.
Re: Benchmarking GPT-4 Turbo – A Cautionary Tale
#16> We designed a test for this theory: we reran the benchmarks without showing the models the instructions to each exercise. Instead, we just told them that they were Exercism exercises, and gave them the exercise names and function stubs. This summarizes all my skepticism agains the AI field. Pretty clear that they aren't solving the problem, they have them memorized.
[deleted]
Re: Benchmarking GPT-4 Turbo – A Cautionary Tale
#17Earlier quoted context omitted.
"They memorized all the problems" is not what was found here and still a wrong overcorrection.
If you are shown only the title of a coding problem and the site name where it's from, and you manage to solve it you are showing that you either cheated or knew the answer.
I mean sure, it memorized some of the answers. I'm not denying that. Clearly, it didn't memorize all of them.
Re: Benchmarking GPT-4 Turbo – A Cautionary Tale
#18> Although the author OCR’ed the SAT questions and believes that they weren’t in the training data
I agree that the author of the tweet fairly underestimates the potential portion of OCR'ed contents in OpenAI's training data. In late August, Nougat[1] is released by Meta, this is an OCR model. Its performance are wild and the model is open source.
I hardly believe that OpenAI does not spend effort on getting more training from OCR content. I also hardly believes that OpenAI waits for a Meta paper to have an internal performant OCR model.
Re: Benchmarking GPT-4 Turbo – A Cautionary Tale
#19> We designed a test for this theory: we reran the benchmarks without showing the models the instructions to each exercise. Instead, we just told them that they were Exercism exercises, and gave them the exercise names and function stubs. This summarizes all my skepticism agains the AI field. Pretty clear that they aren't solving the problem, they have them memorized.
Memorization often gets a bad rap as the underachiever's shortcut. However, it's a fundamental component of any learning process! Our ability to reason, solve problems, and innovate is all built upon a foundation of memorized information. In fact, it's precisely the reason humans have thrived for so long; we were able to memorize and pass down knowledge culturally long before the written word, not because we were 100 times smarter than our nearest cousins. Without memorization, be it in our brains or AI algorithms, there's no foundation to build upon for higher reasoning.
Re: Benchmarking GPT-4 Turbo – A Cautionary Tale
#20Earlier quoted context omitted.
If you are shown only the title of a coding problem and the site name where it's from, and you manage to solve it you are showing that you either cheated or knew the answer.
I mean sure, it memorized some of the answers. I'm not denying that. Clearly, it didn't memorize all of them.
When people say "oh look how amazing, it can solve programming problems!" when in fact it has only seen the models CHEAT, is an enormous problem.
For cases where finding the answer it's perfectly fine, but it's not fine for claims that it can code. There's a huge difference.