I've always been skeptical of benchmarking because of the memorization problem. I recently made up my own (simple) date reasoning benchmark to test this, and found that GPT-4 Turbo actually outperformed GPT-4: https://open.substack.com/pub/talcai/p/making-up-a-new-llm-b...
Benchmarking GPT-4 Turbo – A Cautionary Tale
91–100 of 118 posts
Re: Benchmarking GPT-4 Turbo – A Cautionary Tale
#92I've always been skeptical of benchmarking because of the memorization problem. I recently made up my own (simple) date reasoning benchmark to test this, and found that GPT-4 Turbo actually outperformed GPT-4: https://open.substack.com/pub/talcai/p/making-up-a-new-llm-b...
I like the test but do you take multiple samples / runs of a result? IMO for a proper benchmark you should ask it the same question 10+ times and show a confidence interval, otherwise you don't know if it's just a fluke or a lucky guess.
Re: Benchmarking GPT-4 Turbo – A Cautionary Tale
#93The thing is why does the GPT-4 Turbo and the Updated GPT 3.5 Turbo have only an output of 4,096 tokens? Previous Model: gpt-3.5-turbo-16k, 16385 tokens context and completion (shared) New Model: gpt-3.5-turbo-1106, 16385 tokens context, 4096 tokens completion Previous Model: gpt-4, 8192 tokens context and completion (shared) New Model: gpt-4-1106-preview, 128000 tokens context, 4096 tokens completion Why would the s…
if i'm not mistaken, the model has to be trained for a specific context window
Re: Benchmarking GPT-4 Turbo – A Cautionary Tale
#94I've always been skeptical of benchmarking because of the memorization problem. I recently made up my own (simple) date reasoning benchmark to test this, and found that GPT-4 Turbo actually outperformed GPT-4: https://open.substack.com/pub/talcai/p/making-up-a-new-llm-b...
I like the test but do you take multiple samples / runs of a result? IMO for a proper benchmark you should ask it the same question 10+ times and show a confidence interval, otherwise you don't know if it's just a fluke or a lucky guess.
Ahh good suggestion, I should clarify this in the article. I tried to compensate with volume -- I used a set of 200 questions for the testing. I was using temperature 0, so I'd get the same answer if I ran a single question multiple times.
Re: Benchmarking GPT-4 Turbo – A Cautionary Tale
#95In my day job we use GPT4 quite a bit and we shifted to GPT4 Turbo today. We got a 2-5% performance increase, and quite a bit of speed increase as well. Not to say that the parent post is incorrect, of course. Only that its not as cut and dry as a "GPT4 Turbo is distilled (read: watered down) GPT4".
Interesting. What do you use it for?
Currently only for unstructured (OCR) text to structured text conversion.
We're transitioning from a legacy codebase full of regexes and undocumented functions that are understood only by the developer and god. The developers left and I don't believe in god. We tried throwing the unstructed mess to GPT, alongwith a few examples and got surprisingly good results.
Re: Benchmarking GPT-4 Turbo – A Cautionary Tale
#96> We designed a test for this theory: we reran the benchmarks without showing the models the instructions to each exercise. Instead, we just told them that they were Exercism exercises, and gave them the exercise names and function stubs. This summarizes all my skepticism agains the AI field. Pretty clear that they aren't solving the problem, they have them memorized.
+100 to that. My biggest scepticism is people actually creating a new problem while thinking they are solving problem. Don't get me wrong, translating natural language ideas into code is fun and all, the truth it is also code, yet in ambiguous language format given to the machine. When did natural language became better for expressing development ideas than code? I know – when you don't know how to code in the first…
That sounds a lot like gatekeeping.
These tools will empower folks who aren’t developers to build stuff and maybe learn a bit more about how programming works.
They will enable folks who have ideas, but can’t express them, to actually be able to create what they are imagining.
That’s awesome.
Code isn’t beautiful (except for a few rare exceptions). Creating something with code is.
Re: Benchmarking GPT-4 Turbo – A Cautionary Tale
#97The problem is the discussed results are comparing proportions of a relatively small number - 67 questions. If you model this as a binomial distribution, then 62/67 which GPT4-turbo got gives a 95% confidence interval of the 'true' performance of 83.4% to 97.5%, ie it comfortably includes the proportion that GPT4 achieved (64/67=95.5%). I think the evidence from these tests are not strong enough to draw conclusions f…
I’m not surprised, most people can’t even tell the median from the mean.
Re: Benchmarking GPT-4 Turbo – A Cautionary Tale
#98Earlier quoted context omitted.
But in this case it's not like that at all. They only saw the NAME of the problem. Like if I said "Page 23 of Mathbook Y, problem number 3". Which happens to be 6x6.
If I gave you a programming problem and all I told you was that the problem name was Traveling Salesman, you might be able to solve it based on that. If not that, then if I just said "fizzbuzz" to you, I'm sure you would be able to give the solution without me needing to say any other descriptions of the problem
Again, because of memorization, not being able to code.
Re: Benchmarking GPT-4 Turbo – A Cautionary Tale
#99Earlier quoted context omitted.
Does aider work with c# at all?
Yes! Thanks for asking. I've been meaning to address these kinds of questions in the aider FAQ [0]. Here's the entry I just added: Aider supports pretty much all the popular coding languages. This is partly because GPT-4 is fluent in most mainstream languages, and familiar with popular libraries, packages and frameworks. In fact, coding with aider is sometimes the most magical when you're working in a language that y…
I was actually wondering this myself yesterday. So it's not possible to plug a different tree-sitter implementation in for a niche language?
Re: Benchmarking GPT-4 Turbo – A Cautionary Tale
#100> We designed a test for this theory: we reran the benchmarks without showing the models the instructions to each exercise. Instead, we just told them that they were Exercism exercises, and gave them the exercise names and function stubs. This summarizes all my skepticism agains the AI field. Pretty clear that they aren't solving the problem, they have them memorized.
+100 to that. My biggest scepticism is people actually creating a new problem while thinking they are solving problem. Don't get me wrong, translating natural language ideas into code is fun and all, the truth it is also code, yet in ambiguous language format given to the machine. When did natural language became better for expressing development ideas than code? I know – when you don't know how to code in the first…
Natural language isn't superior to computer languages. NL allows you to describe a software concept in a computer language and framework neutral way. The LLM generates the code. The real benefit is when you work across languages and frameworks. It is difficult to keep all of the details of all of the framework calls in your head all of the time.