Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Benchmarking GPT-4 Turbo – A Cautionary Tale

blog.mentat.ai

Post reply on HN
Print view View original

101–110 of 118 posts

Re: Benchmarking GPT-4 Turbo – A Cautionary Tale

#101

Post by epiccoleman »

anotherpaulg wrote: post #74
strich wrote: post #63

Earlier quoted context omitted.

Does aider work with c# at all?

Yes! Thanks for asking. I've been meaning to address these kinds of questions in the aider FAQ [0]. Here's the entry I just added: Aider supports pretty much all the popular coding languages. This is partly because GPT-4 is fluent in most mainstream languages, and familiar with popular libraries, packages and frameworks. In fact, coding with aider is sometimes the most magical when you're working in a language that y…

I've just started playing with aider this week, and I find it extremely fun and exciting. But I will say that I've had middling results with an Elixir / Phoenix app. I don't think this has anything to do with aider - rather, I think that the GPT models haven't quite internalized the new approaches in Phoenix 1.7, since up until Turbo their training data was fairly old and probably still contains more pre 1.7 Phoenix examples than post 1.7.

In spite of these frustrations, I have had some genuinely amazing moments coding with GPT-4 lately though. I upgraded to ChatGPT plus lately and it's just mindblowing how helpful it can be in the right contexts. I'm hoping that as I get better with aider I might just drop the ChatGPT sub and stick to API usage.

I totally understand the skepticism many have, because this stuff is still a bit finicky - but I'm overwhelmed by a sense of how fucking _cool_ this stuff is quite often.

Re: Benchmarking GPT-4 Turbo – A Cautionary Tale

#102

Post by CamperBob2 »

KaoruAoiShiho wrote: post #8

Why are all the comments here so negative... this is a good thing, turbo has less memorization but keeps the same reasoning ability. That's excellent and a relief.

People here spent a lot of time (and money) in school learning to do things that can now be automated. The whining is just beginning.

Re: Benchmarking GPT-4 Turbo – A Cautionary Tale

#103

Post by anotherpaulg »

lemming wrote: post #99
anotherpaulg wrote: post #74

Earlier quoted context omitted.

Yes! Thanks for asking. I've been meaning to address these kinds of questions in the aider FAQ [0]. Here's the entry I just added: Aider supports pretty much all the popular coding languages. This is partly because GPT-4 is fluent in most mainstream languages, and familiar with popular libraries, packages and frameworks. In fact, coding with aider is sometimes the most magical when you're working in a language that y…

I was actually wondering this myself yesterday. So it's not possible to plug a different tree-sitter implementation in for a niche language?

It should be possible, but not currently. Aider would need a bit more configurability to be able to load up arbitrary tree-sitter language implementations at runtime.

There's an open issue you might want to follow for updates:

https://github.com/paul-gauthier/aider/issues/321

Re: Benchmarking GPT-4 Turbo – A Cautionary Tale

#104

Post by hfjjbf »

anotherpaulg wrote: post #80
biobootloader wrote: post #69

Earlier quoted context omitted.

Hey Paul, I'm a Mentat author. > I also notice that the instructions prompt that mentat uses seems to be inspired by the aider benchmark? Glad to see others adopting similar benchmarking approaches. We were inspired by you to use Exercism as a benchmark, thank you! We will add attribution for that. We switched our original instruction prompts for that benchmark to be similar to Aiders to allow for fair comparison. >…

I didn't spend much time looking, but your benchmark prompting inspired me to search your repo for "aider". The results were 3 PRs where aider was mentioned in the conversations [0]. The "code map" PR in particular mentions being "inspired by aider", links to aider and seems to include a bunch of code from aider's old ctags based "repo map" implementation. This isn't an insignificant component of an AI coding tool. A…

[dead]

Re: Benchmarking GPT-4 Turbo – A Cautionary Tale

#105

Post by refulgentis »

qup wrote: post #93
jessenaser wrote: post #73

The thing is why does the GPT-4 Turbo and the Updated GPT 3.5 Turbo have only an output of 4,096 tokens? Previous Model: gpt-3.5-turbo-16k, 16385 tokens context and completion (shared) New Model: gpt-3.5-turbo-1106, 16385 tokens context, 4096 tokens completion Previous Model: gpt-4, 8192 tokens context and completion (shared) New Model: gpt-4-1106-preview, 128000 tokens context, 4096 tokens completion Why would the s…

if i'm not mistaken, the model has to be trained for a specific context window

More or less, like there's stuff you can do to extend the window of an existing model fairly easily, i.e. LoRA type training budget, O($1000).

But in practice, even when context_size max output token count was enabled, it simply couldn't make use of it, no matter how many prompt engineering tricks I threw at it.[1] And I've heard anecdotally that it's true for that LoRA-type technique.

[1] TL;DR, about 1/5th the actual length: write 100 pages, 3 paragraphs each, number the pages as you go and write 1 page at a time until 100. Also write out "I have written page N and need to write 100 pages total" after each page.

Inevitably it would "get tired" and be like "end page 23...now page 100"

Re: Benchmarking GPT-4 Turbo – A Cautionary Tale

#106

Post by FeepingCreature »

boxed wrote: post #98
stnmtn wrote: post #60

Earlier quoted context omitted.

If I gave you a programming problem and all I told you was that the problem name was Traveling Salesman, you might be able to solve it based on that. If not that, then if I just said "fizzbuzz" to you, I'm sure you would be able to give the solution without me needing to say any other descriptions of the problem

Again, because of memorization, not being able to code.

But in that case, not memorization of the specific problem set, but "programming background knowledge." Hardly something to blame the machine for when we rely on it every day.

Re: Benchmarking GPT-4 Turbo – A Cautionary Tale

#107

Post by ChatGTP »

rocgf wrote: post #59
ChatGTP wrote: post #57

Earlier quoted context omitted.

The human brain is a model?

It models the world around it, so it's fairly similar to what GPT does, especially with the newly-added image capabilities and stuff.

But the brain itself is not a model.

Re: Benchmarking GPT-4 Turbo – A Cautionary Tale

#108

Post by attentive »

geraltofrivia wrote: post #95
hu3 wrote: post #81

Earlier quoted context omitted.

Interesting. What do you use it for?

Currently only for unstructured (OCR) text to structured text conversion. We're transitioning from a legacy codebase full of regexes and undocumented functions that are understood only by the developer and god. The developers left and I don't believe in god. We tried throwing the unstructed mess to GPT, alongwith a few examples and got surprisingly good results.

> undocumented functions that are understood only by the developer and god

oh the irony :)

Re: Benchmarking GPT-4 Turbo – A Cautionary Tale

#109

Post by geraltofrivia »

attentive wrote: post #108
geraltofrivia wrote: post #95

Earlier quoted context omitted.

Currently only for unstructured (OCR) text to structured text conversion. We're transitioning from a legacy codebase full of regexes and undocumented functions that are understood only by the developer and god. The developers left and I don't believe in god. We tried throwing the unstructed mess to GPT, alongwith a few examples and got surprisingly good results.

> undocumented functions that are understood only by the developer and god oh the irony :)

I don't follow . Ironical how?

Re: Benchmarking GPT-4 Turbo – A Cautionary Tale

#110

Post by gloosx »

ericrallen wrote: post #96
gloosx wrote: post #84

Earlier quoted context omitted.

+100 to that. My biggest scepticism is people actually creating a new problem while thinking they are solving problem. Don't get me wrong, translating natural language ideas into code is fun and all, the truth it is also code, yet in ambiguous language format given to the machine. When did natural language became better for expressing development ideas than code? I know – when you don't know how to code in the first…

That sounds a lot like gatekeeping. These tools will empower folks who aren’t developers to build stuff and maybe learn a bit more about how programming works. They will enable folks who have ideas, but can’t express them, to actually be able to create what they are imagining. That’s awesome. Code isn’t beautiful (except for a few rare exceptions). Creating something with code is.

I agree it is a great tool for learning, but I don't believe anything more complex or of real use can be made AND maintained with it.
Post reply on HN