Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Claude 4

anthropic.com

Post reply on HN
Print view View original

531–540 of 1001 posts

Re: Claude 4

#531

Post by BeetleB »

GolDDranks wrote: post #437

After using Claude 3.7 Sonnet for a few weeks, my verdict is that its coding abilities are unimpressive both for unsupervised coding but also for problem solving/debugging if you are expecting accurate results and correct code. However, as a debugging companion, it's slightly better than a rubber duck, because at least there's some suspension of disbelief so I tend to explain things to it earnestly and because of tha…

I've noticed an interesting trend:

Most people who are happy with LLM coding say something like "Wow, it's awesome. I asked it to do X and it did it so fast with minimal bugs, and good code", and occasionally show the output. Many provide even more details.

Most people who are not happy with LLM coding ... provide almost no details.

As someone who's impressed by LLM coding, when I read a post like yours, I tend to have a lot of questions, and generally the post doesn't have the answers.

1. What type of problem did you try it out with?

2. Which model did you use (you get points for providing that one!)

3. Did you consider a better model (compare how Gemini 2.5 Pro compares to Sonnet 3.7 on the Aider leaderboard)?

4. What were its failings? Buggy code? Correct code but poorly architected? Correct code but used some obscure method to solve it rather than a canonical one?

5. Was it working on an existing codebase or was this new code?

6. Did you manage well how many tokens were sent? Did you use a tool that informs you of the number of tokens for each query?

7. Which tool did you use? It's not just a question of the model, but of how the tool handles the prompts/agents under it. Aider is different from Code which is different from Cursor which is different form Windsurf.

8. What strategy did you follow? Did you give it the broad spec and ask it to do anything? Did you work bottom up and work incrementally?

I'm not saying LLM coding is the best or can replace a human. But for certain use cases (e.g. simple script, written from scratch), it's absolutely fantastic. I (mostly) don't use it on production code, but little peripheral scripts I need to write (at home or work), it's great. And that's why people like me wonder what people like you are doing differently.

But such people aren't forthcoming with the details.

Re: Claude 4

#532

Post by _peregrine_ »

jjwiseman wrote: post #494
_peregrine_ wrote: post #396

Already test Opus 4 and Sonnet 4 in our SQL Generation Benchmark ( https://llm-benchmark.tinybird.live/ ) Opus 4 beat all other models. It's good.

Please add GPT o3.

Noted, also feel free to add an issue to the GitHub repo: https://github.com/tinybirdco/llm-benchmark

Re: Claude 4

#533

Post by strangescript »

minimaxir wrote: post #102
tptacek wrote: post #83

Have they documented the context window changes for Claude 4 anywhere? My (barely informed) understanding was one of the reasons Gemini 2.5 has been so useful is that it can handle huge amounts of context --- 50-70kloc?

Context window is unchanged for Sonnet. (200k in/64k out): https://docs.anthropic.com/en/docs/about-claude/models/overv... In practice, the 1M context of Gemini 2.5 isn't that much of a differentiator because larger context has diminishing returns on adherence to later tokens.

Yeah, but why aren't they attacking that problem? Is it just impossible, because it would be a really simple win with regards to coding. I am huge enthusiast, but I am starting to feel a peak.

Re: Claude 4

#534

Post by _peregrine_ »

varunneal wrote: post #475
_peregrine_ wrote: post #396

Already test Opus 4 and Sonnet 4 in our SQL Generation Benchmark ( https://llm-benchmark.tinybird.live/ ) Opus 4 beat all other models. It's good.

Why is o3-mini there but not o3?

We should definitely add o3 - probably will soon. Also looking at testing the Qwen models

Re: Claude 4

#535

Post by _peregrine_ »

mritchie712 wrote: post #493
_peregrine_ wrote: post #396

Already test Opus 4 and Sonnet 4 in our SQL Generation Benchmark ( https://llm-benchmark.tinybird.live/ ) Opus 4 beat all other models. It's good.

looks like this is one-shot generation right? I wonder how much the results would change with a more agentic flow (e.g. allow it to see an error or select * from the_table first). sonnet seems particularly good at in-session learning (e.g. correcting it's own mistakes based on a linter).

Actually no, we have it up to 3 attempts. In fact, Opus 4 failed on 36/50 tests on the first attempt, but it was REALLY good at nailing the second attempt after receiving error feedback.

Re: Claude 4

#537

Post by hnlmorg »

paulddraper wrote: post #497
bredren wrote: post #270

Earlier quoted context omitted.

It still matters for software packages. Particularly python packages that have to do with programming with AI! They are evolving quickly, with deprecation and updated documentation. Having to correct for this in system prompts is a pain. It would be great if the models were updating portions of their content more recently than others. For the tailwind example in parent-sibling comment, should absolutely be as up to d…

How often are base level libraries/frameworks changing in incomparable ways?

That depends on the language and domain.

MCP itself isn’t even a year old.

Re: Claude 4

#538

Post by _peregrine_ »

Workaccount2 wrote: post #460
_peregrine_ wrote: post #396

Already test Opus 4 and Sonnet 4 in our SQL Generation Benchmark ( https://llm-benchmark.tinybird.live/ ) Opus 4 beat all other models. It's good.

This is a pretty interesting benchmark because it seems to break the common ordering we see with all the other benchmarks.

Yeah I mean SQL is pretty nuanced - one of the things we want to improve in the benchmark is how we measure "success", in the sense that multiple correct SQL results can look structurally dissimilar while semantically answering the prompt.

There's some interesting takeaways we learned here after the first round: https://www.tinybird.co/blog-posts/we-graded-19-llms-on-sql-...

Re: Claude 4

#539

Post by make3 »

Doohickey-d wrote: post #48

> Users requiring raw chains of thought for advanced prompt engineering can contact sales So it seems like all 3 of the LLM providers are now hiding the CoT - which is a shame, because it helped to see when it was going to go down the wrong track, and allowing to quickly refine the prompt to ensure it didn't. In addition to openAI, Google also just recently started summarizing the CoT, replacing it with an, in my opi…

it just makes it too easy to distill the reasoning into a separate model I guess. though I feel like o3 shows useful things about the reasoning while it's happening

Re: Claude 4

#540

Post by wrsh07 »

swyx wrote: post #171
wrsh07 wrote: post #162

Earlier quoted context omitted.

My understanding for the original OpenAI and anthropic labels was essentially: gpt2 was 100x more compute than gpt1. Same for 2 to 3. Same for 3 to 4. Thus, gpt 4.5 was 10x more compute^ If anthropic is doing the same thing, then 3.5 would be 10x more compute vs 3. 3.7 might be 3x more than 3.5. and 4 might be another ~3x. ^ I think this maybe involves words like "effective compute", so yeah it might not be a full pr…

beyond 4 thats no longer true - marketing took over from the research

Oh shoot I thought that still applied to 4.5 just in a more "effective compute" way (not 100x more parameters, but 100x more compute in training)

But alas, it's not like 3nm fab means the literal thing either. Marketing always dominates (and not necessarily in a way that adds clarity)

Post reply on HN