Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Benchmarking GPT-4 Turbo – A Cautionary Tale

blog.mentat.ai

Post reply on HN
Print view View original

111–118 of 118 posts

Re: Benchmarking GPT-4 Turbo – A Cautionary Tale

#111

Post by [deleted] »

doug_durham wrote: post #100
gloosx wrote: post #84

Earlier quoted context omitted.

+100 to that. My biggest scepticism is people actually creating a new problem while thinking they are solving problem. Don't get me wrong, translating natural language ideas into code is fun and all, the truth it is also code, yet in ambiguous language format given to the machine. When did natural language became better for expressing development ideas than code? I know – when you don't know how to code in the first…

Natural language isn't superior to computer languages. NL allows you to describe a software concept in a computer language and framework neutral way. The LLM generates the code. The real benefit is when you work across languages and frameworks. It is difficult to keep all of the details of all of the framework calls in your head all of the time.

[deleted]

Re: Benchmarking GPT-4 Turbo – A Cautionary Tale

#112

Post by gloosx »

doug_durham wrote: post #100
gloosx wrote: post #84

Earlier quoted context omitted.

+100 to that. My biggest scepticism is people actually creating a new problem while thinking they are solving problem. Don't get me wrong, translating natural language ideas into code is fun and all, the truth it is also code, yet in ambiguous language format given to the machine. When did natural language became better for expressing development ideas than code? I know – when you don't know how to code in the first…

Natural language isn't superior to computer languages. NL allows you to describe a software concept in a computer language and framework neutral way. The LLM generates the code. The real benefit is when you work across languages and frameworks. It is difficult to keep all of the details of all of the framework calls in your head all of the time.

Where is the evidence for that? Any real-world application made and running by describing software concepts to an LLM?

It is what it is – a novel search engine, lossy and non-credible. Effectively useless on codebases that extend beyond its fairly limited context

Re: Benchmarking GPT-4 Turbo – A Cautionary Tale

#113

Post by ericrallen »

gloosx wrote: post #110
ericrallen wrote: post #96

Earlier quoted context omitted.

That sounds a lot like gatekeeping. These tools will empower folks who aren’t developers to build stuff and maybe learn a bit more about how programming works. They will enable folks who have ideas, but can’t express them, to actually be able to create what they are imagining. That’s awesome. Code isn’t beautiful (except for a few rare exceptions). Creating something with code is.

I agree it is a great tool for learning, but I don't believe anything more complex or of real use can be made AND maintained with it.

I think we’re probably way to early in the AI lifecycle to really form any strongly held beliefs yet.

In the 11 months since ChatGPT was released, things have come a long way. Who knows where we’ll be in another 11 months.

Re: Benchmarking GPT-4 Turbo – A Cautionary Tale

#114

Post by throwaway675309 »

anotherpaulg wrote: post #12

Aider has had an Exercism benchmarking suite for quite some time. Interestingly, my benchmark results of GPT 4 Turbo show an opposite result: the new gpt-4-1106-preview did significantly better on the first try than the March and June models. https://aider.chat/docs/benchmarks-1106.html Aider benchmarks against the 133 Exercism python exercises, not js exercises that mentat's benchmark uses. So this is not an apples-…

I feel like I see this A LOT these days. If you do a Show HN (for example) and your project is directly inspired by somebody else's who came before you, the least you can do is give nominal attribution.

What is it about software development in particular that makes people so seemingly ethically unfettered by blatant plagiarism?

Re: Benchmarking GPT-4 Turbo – A Cautionary Tale

#115

Post by rldjbpin »

boxed wrote: post #4

> We designed a test for this theory: we reran the benchmarks without showing the models the instructions to each exercise. Instead, we just told them that they were Exercism exercises, and gave them the exercise names and function stubs. This summarizes all my skepticism agains the AI field. Pretty clear that they aren't solving the problem, they have them memorized.

if the benchmark means replicating the experience of taking technical interviews by most people, then this is a spot on approach and serves the potential user right.

Re: Benchmarking GPT-4 Turbo – A Cautionary Tale

#116

Post by attentive »

geraltofrivia wrote: post #109
attentive wrote: post #108

Earlier quoted context omitted.

> undocumented functions that are understood only by the developer and god oh the irony :)

I don't follow . Ironical how?

You replaced it with the system that is even worse than "undocumented functions that are understood only by the developer and god" by design. It's not even deterministic.

Re: Benchmarking GPT-4 Turbo – A Cautionary Tale

#117

Post by geraltofrivia »

attentive wrote: post #116
geraltofrivia wrote: post #109

Earlier quoted context omitted.

I don't follow . Ironical how?

You replaced it with the system that is even worse than "undocumented functions that are understood only by the developer and god" by design. It's not even deterministic.

Oh yeah. One hundred percent true :D It just happens to be significantly better both in terms of precision and recall than the former solution.

Re: Benchmarking GPT-4 Turbo – A Cautionary Tale

#118

Post by gloosx »

ericrallen wrote: post #113
gloosx wrote: post #110

Earlier quoted context omitted.

I agree it is a great tool for learning, but I don't believe anything more complex or of real use can be made AND maintained with it.

I think we’re probably way to early in the AI lifecycle to really form any strongly held beliefs yet. In the 11 months since ChatGPT was released, things have come a long way. Who knows where we’ll be in another 11 months.

What I'm trying to say is that the problem is not approachable this way at all – efficiently generating code by describing what you want, since when you compress what you want into a prompt you lose the details, and in order to restore all of them you will need a much bigger prompt volume than code generated. Because it is code itself which compresses an idea but no idea can compress the code well enough. In another 11 months it will be exactly in the same spot - it will not be able to be more efficient at this task by the nature of it.
Post reply on HN