Benchmarking GPT-4 Turbo – A Cautionary Tale • Hacker News Board

Re: Benchmarking GPT-4 Turbo – A Cautionary Tale

#71

Post by thorax » Thu, Nov 09, 2023, 4:11 PM UTC

Havoc wrote: post #34

Is it know what exactly OpenAI does in the background when they make these turbo editions? Seems like sacrificing some quality for large gains on speed and cost but anyone know more detail?

Don't think so, but there were some guesses on 3.5-turbo-- i.e. training a much smaller model on quality questions/answers from GPT-4. Same tactic worked again and again for other LLMs.

I'm definitely curious on the context window increase-- I'm having a hard time telling if it's 'real' vs a fast specially trained summarization prework step. That being said, it's been doing a rather solid job not losing info in that context window in my minor anecdotal use cases.

Re: Benchmarking GPT-4 Turbo – A Cautionary Tale

#72

Post by ren_engineer » Thu, Nov 09, 2023, 4:14 PM UTC

boxed wrote: post #4

> We designed a test for this theory: we reran the benchmarks without showing the models the instructions to each exercise. Instead, we just told them that they were Exercism exercises, and gave them the exercise names and function stubs. This summarizes all my skepticism agains the AI field. Pretty clear that they aren't solving the problem, they have them memorized.

even if it's not true AI or even an architecture with the potential to become AI, LLMs are already good enough to provide real world value. Obviously "super autocomplete" isn't as sexy as true AI, but still very useful

Re: Benchmarking GPT-4 Turbo – A Cautionary Tale

#73

Post by jessenaser » Thu, Nov 09, 2023, 4:15 PM UTC

The thing is why does the GPT-4 Turbo and the Updated GPT 3.5 Turbo have only an output of 4,096 tokens?

Previous Model: gpt-3.5-turbo-16k, 16385 tokens context and completion (shared)

New Model: gpt-3.5-turbo-1106, 16385 tokens context, 4096 tokens completion

Previous Model: gpt-4, 8192 tokens context and completion (shared)

New Model: gpt-4-1106-preview, 128000 tokens context, 4096 tokens completion

Why would the same size of a 16K GPT-3.5 model now not allow larger completion sizes?

Why would the new GPT-4 reduce the completion tokens as well, gpt-4 can do 8192 and gpt-4-32k can do 32768 completion tokens. Now the limit is 4096.

So you would need to change the way you prompt (split the responses) to be able to get a longer response.

---

So are these new models taking the old base models of 4K tokens context and completion and changing the context to 128000 but leaving the completion the same? If they could get gpt-4 to have gpt-4-8k and gpt-4-32k, why couldn't have it been 128000 context and 32768 completion?

Re: Benchmarking GPT-4 Turbo – A Cautionary Tale

#74

Post by anotherpaulg » Thu, Nov 09, 2023, 4:16 PM UTC

strich wrote: post #63

anotherpaulg wrote: post #12

Aider has had an Exercism benchmarking suite for quite some time. Interestingly, my benchmark results of GPT 4 Turbo show an opposite result: the new gpt-4-1106-preview did significantly better on the first try than the March and June models. https://aider.chat/docs/benchmarks-1106.html Aider benchmarks against the 133 Exercism python exercises, not js exercises that mentat's benchmark uses. So this is not an apples-…

Does aider work with c# at all?

Yes!

Thanks for asking. I've been meaning to address these kinds of questions in the aider FAQ [0]. Here's the entry I just added:

Aider supports pretty much all the popular coding languages. This is partly because GPT-4 is fluent in most mainstream languages, and familiar with popular libraries, packages and frameworks.

In fact, coding with aider is sometimes the most magical when you're working in a language that you are less familiar with. GPT often knows the language better than you, and can generate all the boilerplate to get to the heart of your problem. GPT will often solve your problem in an elegant way using a library or package that you weren't even aware of.

Aider uses tree-sitter to do code analysis and help GPT navigate larger code bases by producing a repository map [1].

Aider can currently produce repository maps for most mainstream languages, listed below. But aider should work quite well for other languages, even without repo map support.

  - C
  - C#
  - C++
  - Emacs Lisp
  - Elixir
  - Elm
  - Go
  - Java
  - Javascript
  - OCaml
  - PHP
  - Python
  - QL
  - Ruby
  - Rust
  - Typescript

[0] https://aider.chat/docs/faq.html#what-code-languages-does-ai...

[1] https://aider.chat/docs/repomap.html

Re: Benchmarking GPT-4 Turbo – A Cautionary Tale

#75

Post by cultureswitch » Thu, Nov 09, 2023, 4:24 PM UTC

Everyone who has any knowledge of machine learning knows that you don't evaluate your model by testing it on parts of its training data. The issue is, the training data for GPT-4 appears to be "everything".

Re: Benchmarking GPT-4 Turbo – A Cautionary Tale

#76

Post by epups » Thu, Nov 09, 2023, 4:26 PM UTC

jphoward wrote: post #53

The problem is the discussed results are comparing proportions of a relatively small number - 67 questions. If you model this as a binomial distribution, then 62/67 which GPT4-turbo got gives a 95% confidence interval of the 'true' performance of 83.4% to 97.5%, ie it comfortably includes the proportion that GPT4 achieved (64/67=95.5%). I think the evidence from these tests are not strong enough to draw conclusions f…

Yes. I see people make this mistake time and again when evaluating LLMs. For a proper comparison, it's not enough to simply throw less than a hundred questions at it and point to a single digit difference. Not to mention that LLMs have some inherent randomness, so even if you passed the exact same tasks to the same model you would expect some variance.

I see a lot of room of improvement in how we apply statistics to understanding LLM performance.

Re: Benchmarking GPT-4 Turbo – A Cautionary Tale

#77

Post by ja3k » Thu, Nov 09, 2023, 4:34 PM UTC

anotherpaulg wrote: post #12

Aider has had an Exercism benchmarking suite for quite some time. Interestingly, my benchmark results of GPT 4 Turbo show an opposite result: the new gpt-4-1106-preview did significantly better on the first try than the March and June models. https://aider.chat/docs/benchmarks-1106.html Aider benchmarks against the 133 Exercism python exercises, not js exercises that mentat's benchmark uses. So this is not an apples-…

Sorry about that. We updated the blog with attribution and put an attributing comment in our code base where we use your benchmarking prompts. We'll probably delete our implementation of your response format later today since we just had it for benchmarking.

Re: Benchmarking GPT-4 Turbo – A Cautionary Tale

#78

Post by drcode » Thu, Nov 09, 2023, 4:46 PM UTC

boxed wrote: post #4

> We designed a test for this theory: we reran the benchmarks without showing the models the instructions to each exercise. Instead, we just told them that they were Exercism exercises, and gave them the exercise names and function stubs. This summarizes all my skepticism agains the AI field. Pretty clear that they aren't solving the problem, they have them memorized.

You can call it whatever you want, all I know is I used to write programs in lines of code, then blocks of code at a time, spit out by LLMs

Using GPT-4 Turbo yesterday, I feel like I'm moving to pages of code at a time now.

Taking the ideas in my head and turning them into reality is so easy now

Re: Benchmarking GPT-4 Turbo – A Cautionary Tale

#79

Post by driverdan » Thu, Nov 09, 2023, 4:51 PM UTC

This is a big problem with independent LLM testing. You need to make sure your test set isn't included in the training set which isn't easy with closed source models.

This makes me think of how hardware manufacturers optimize for benchmarks. Closed source LLMs can intentionally include likely test data in their training set to artificially inflate results. I'm not saying they are intentionally doing that now, but they could.

Re: Benchmarking GPT-4 Turbo – A Cautionary Tale

#80

Post by anotherpaulg » Thu, Nov 09, 2023, 4:51 PM UTC

biobootloader wrote: post #69

anotherpaulg wrote: post #12

Aider has had an Exercism benchmarking suite for quite some time. Interestingly, my benchmark results of GPT 4 Turbo show an opposite result: the new gpt-4-1106-preview did significantly better on the first try than the March and June models. https://aider.chat/docs/benchmarks-1106.html Aider benchmarks against the 133 Exercism python exercises, not js exercises that mentat's benchmark uses. So this is not an apples-…

Hey Paul, I'm a Mentat author. > I also notice that the instructions prompt that mentat uses seems to be inspired by the aider benchmark? Glad to see others adopting similar benchmarking approaches. We were inspired by you to use Exercism as a benchmark, thank you! We will add attribution for that. We switched our original instruction prompts for that benchmark to be similar to Aiders to allow for fair comparison. >…

I didn't spend much time looking, but your benchmark prompting inspired me to search your repo for "aider". The results were 3 PRs where aider was mentioned in the conversations [0].

The "code map" PR in particular mentions being "inspired by aider", links to aider and seems to include a bunch of code from aider's old ctags based "repo map" implementation. This isn't an insignificant component of an AI coding tool.

Aider is open source and I try and share my learnings as I'm building it. So it's great when other projects get inspiration from aider! But it is polite to provide attribution for such inspiration, especially if you crib from code with an attribution license.

[0] https://github.com/search?q=repo%3AAbanteAI%2Fmentat+aider&t...