Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GPT-5.6, Grok 4.5, Claude, and Muse Spark build the same 4 apps

tryai.dev

Post reply on HN
Print view View original

61–70 of 95 posts

Re: GPT-5.6, Grok 4.5, Claude, and Muse Spark build the same 4 apps

#61

Post by tedsanders »

thebigspacefuck wrote: post #12

(LM)Arena is basically this. IMO it’s the best benchmark that avoids benchmaxxing Agent: https://arena.ai/leaderboard/agent Web dev: https://arena.ai/leaderboard/code/webdev Currently Fable and 5.6 are neck and neck on web dev which is basically the same finding as this.

Arena can definitely be benchmaxxed a bit, if you try. The distribution of prompts there is very different than usage by regular coders. E.g., lots of requests for one-shot games from scratch. So if you fine-tuned your model to be great at making fun one-shot games from underspecified prompts, your coding model might look better than it is (on general tasks, at least).

I work at OpenAI, and am happy to say we don't try to juice our scores here, as doing so would be counterproductive and make Arena a worse signal for everyone.

Re: GPT-5.6, Grok 4.5, Claude, and Muse Spark build the same 4 apps

#63

Post by mikeocool »

platinumrad wrote: post #36

Maybe I'm a control freak, but asking agents to one-shot random apps is nothing like how I actually use AI in software engineering.

Yeah, the models have all been really good at generating greenfield apps for a really long time (in the scope of LLM time).

I suppose it’s interesting to see how they make better greenfield apps. But I am much more interested in how they solve hard problems in existing gnarly codebases.

Re: GPT-5.6, Grok 4.5, Claude, and Muse Spark build the same 4 apps

#64

Post by razighter777 »

paxys wrote: post #24

> Separate question, separate table. This is our standard latency harness (three short prompts, five reps, 400-token cap), not the build tasks. tok/s is output tokens over wall-clock, uniform for all. > so their tok/s is a ceiling, not a true decode rate. The clear read: the GPT-5.6 tiers are the snappiest models here on short prompts (Luna answers in about a second), Qwen is absurdly cheap and fast, and DeepSeek and…

It's so telling and offputting.

Please write like a normal human and put the effort in to type what you want to say. Using AI to make your writing is not only lazy, it's bland, tiresome, and disrespectful of the reader's time.

Re: GPT-5.6, Grok 4.5, Claude, and Muse Spark build the same 4 apps

#65

Post by drivebyhooting »

nomel wrote: post #51
rbehrends wrote: post #17

My concern with most of these visual benchmarks, popular as they are, is that they are likely more indicative of knowledge (i.e. how comprehensive the training data is and how well it can be retrieved from the model) than of reasoning ability. I don't see in particular how a model would construct a CoT that mapped somehow to a representation of the cube geometry and its animations in latent space without a large chun…

> without a large chunk of that being pre-existing information. Is there any evidence that novel reasoning is present in LLM? I've never been able to make that work, and I believe Apple's paper some time ago was good evidence that it doesn't exist. In my experience, sparse latent spaces result in a complete, comical, failure in reasoning.

See the new mathematical proof published by OpenAI.

I’m not very valiant to verify its veracity. But even if the math is merely derivative it merits mention.

Re: GPT-5.6, Grok 4.5, Claude, and Muse Spark build the same 4 apps

#66

Post by dang »

minimaxir wrote: post #14
ttoinou wrote: post #6

"This isn't objective." Correct, and we are not pretending it is. We are not handing down a scientific verdict. Actually, you are doing rational investigation in a fuzzy probabilistic new/emergent space, with open sharing to the world. I don’t understand why people downplay themselves and put on a pedestal others supposedly serious sciences.

It's a preemptive defense against methodology cynicism seen often on sites including but not limited to Hacker News. I've been guilty of including such defenses myself over the years because I've gotten annoyed with receiving such cynicism. Look at the top comment on their previous HN submission: https://news.ycombinator.com/item?id=48839886

Ugh, that is bad, and I'm sorry I didn't see it at the top of the thread yesterday.

Really, the deeper problem is the upvotes that cause such posts to rise to the top of a thread and stick there, drowning out curious conversation and giving people a bad impression of the entire community. Unfortunately, the upvote problem seems basically unsolvable - people don't do it consciously and it's very much a tragedy-of-the-commons problem. So we're stuck with moderation on the comments.

Re: GPT-5.6, Grok 4.5, Claude, and Muse Spark build the same 4 apps

#67

Post by sublinear »

fluidcruft wrote: post #38
paxys wrote: post #24

> Separate question, separate table. This is our standard latency harness (three short prompts, five reps, 400-token cap), not the build tasks. tok/s is output tokens over wall-clock, uniform for all. > so their tok/s is a ceiling, not a true decode rate. The clear read: the GPT-5.6 tiers are the snappiest models here on short prompts (Luna answers in about a second), Qwen is absurdly cheap and fast, and DeepSeek and…

Yeah, I can't figure out where this "voice" comes from and it is so impossible to get rid of. It is so grating.

It's the unholy trinity of pundits, marketing, and HR.

Their writing was already painted into this corner long before the LLM epoch and they continue to publish more than anyone else.

Re: GPT-5.6, Grok 4.5, Claude, and Muse Spark build the same 4 apps

#68

Post by didip »

I actually like this methodology of testing AI much better than all the other benchmark tests.

Real world is messy, other benchmarks are clearly gameable by the Chinese open models.

Great job! And I don’t care about the tone of the article, it’s readable just fine.

Re: GPT-5.6, Grok 4.5, Claude, and Muse Spark build the same 4 apps

#69

Post by [deleted] »

razighter777 wrote: post #64
paxys wrote: post #24

> Separate question, separate table. This is our standard latency harness (three short prompts, five reps, 400-token cap), not the build tasks. tok/s is output tokens over wall-clock, uniform for all. > so their tok/s is a ceiling, not a true decode rate. The clear read: the GPT-5.6 tiers are the snappiest models here on short prompts (Luna answers in about a second), Qwen is absurdly cheap and fast, and DeepSeek and…

It's so telling and offputting. Please write like a normal human and put the effort in to type what you want to say. Using AI to make your writing is not only lazy, it's bland, tiresome, and disrespectful of the reader's time.

[deleted]

Re: GPT-5.6, Grok 4.5, Claude, and Muse Spark build the same 4 apps

#70

Post by nomel »

drivebyhooting wrote: post #65
nomel wrote: post #51

Earlier quoted context omitted.

> without a large chunk of that being pre-existing information. Is there any evidence that novel reasoning is present in LLM? I've never been able to make that work, and I believe Apple's paper some time ago was good evidence that it doesn't exist. In my experience, sparse latent spaces result in a complete, comical, failure in reasoning.

See the new mathematical proof published by OpenAI. I’m not very valiant to verify its veracity. But even if the math is merely derivative it merits mention.

True, but that's an unknown internal model, without details of the architecture. We'll have to see if the LLM model, itself, was responsible for the "novel" bits, or if it was stuffs bolted onto the LLM that made it possible. I suppose "LLM" is maybe no longer sufficient to describe the systems that LLM are being integrated into, so maybe my point is pedantic/semantic.
Post reply on HN