Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Claude 4

anthropic.com

Post reply on HN
Print view View original

771–780 of 1001 posts

Re: Claude 4

#771

Post by IAmGraydon »

SamBam wrote: post #545

This is the first LLM that has been able to answer my logic puzzle on the first try without several minutes of extended reasoning. > A man wants to cross a river, and he has a cabbage, a goat, a wolf and a lion. If he leaves the goat alone with the cabbage, the goat will eat it. If he leaves the wolf with the goat, the wolf will eat it. And if he leaves the lion with either the wolf or the goat, the lion will eat the…

Now that you've posted this online, you can consider it unusable as a test. If you've ever posted it online before now, you can consider the results null and void.

Beyond that, as others have mentioned, this is not actually a logic puzzle at all, as there are multiple correct answers.

Re: Claude 4

#772

Post by elcritch »

BoredPositron wrote: post #630
Doohickey-d wrote: post #48

> Users requiring raw chains of thought for advanced prompt engineering can contact sales So it seems like all 3 of the LLM providers are now hiding the CoT - which is a shame, because it helped to see when it was going to go down the wrong track, and allowing to quickly refine the prompt to ensure it didn't. In addition to openAI, Google also just recently started summarizing the CoT, replacing it with an, in my opi…

Because it's alchemy and everyone believes they have an edge on turning lead into gold.

I've been thinking for a couple of months now that prompt engineering, and therefore CoT, is going to become the "secret sauce" companies want to hold onto.

If anything that is where the day to day pragmatic engineering gets done. Like with early chemistry, we didn't need to precisely understand chemical theory to produce mass industrial processes by making a good enough working model, some statistical parameters, and good ole practical experience. People figured out steel making and black powder with alchemy.

The only debate now is whether the prompt engineering models are currently closer to alchemy or modern chemistry? I'd say we're at advanced alchemy with some hints of rudimentary chemistry.

Also, unrelated but with CERN turning lead into gold, doesn't that mean the alchemists were correct, just fundamentally unprepared for the scale of the task? ;)

Re: Claude 4

#773

Post by knowaveragejoe »

Falimonda wrote: post #283
thimabi wrote: post #67

It’s been hard to keep up with the evolution in LLMs. SOTA models basically change every other week, and each of them has its own quirks. Differences in features, personality, output formatting, UI, safety filters… make it nearly impossible to migrate workflows between distinct LLMs. Even models of the same family exhibit strikingly different behaviors in response to the same prompt. Still, having to find each model’…

Have you tried a package like LiteLLM so that you can more easily validate and switch to a newer model? The key seems to be in curating your application's evaluation set.

I'd love something like litellm, but simpler. I'm not provisioning models for my organization, I don't need to granularly track spend, I just want one endpoint to point every tool or client at for ease of configuration and curiosity around usage.

Re: Claude 4

#774

Post by adamtaylor_13 »

dbingham wrote: post #743

It feels like these new models are no longer making order of magnitude jumps, but are instead into the long tail of incremental improvements. It seems like we might be close to maxing out what the current iteration of LLMs can accomplish and we're into the diminishing returns phase. If that's the case, then I have a bad feeling for the state of our industry. My experience with LLMs is that their code does _not_ cut i…

That’s funny, my experience has been the exact opposite.

Claude Code has single-handedly 2-3x my coding productivity. I haven’t even used Claude 4 yet so I’m pretty excited to try it out.

But even trusty ol 3.7 is easily helping me out out 2-3x the amount of code I was before. And before anyone asks, yes it’s all peer-reviewed and I read every single line.

It’s been an absolute game changer.

Also to your point about most engineering being thinking: I can test 4-5 ideas in the time it took me to test a single idea in the last. And once you find the right idea, it 100% codes faster than you do.

Re: Claude 4

#775

Post by hodgehog11 »

dbingham wrote: post #743

It feels like these new models are no longer making order of magnitude jumps, but are instead into the long tail of incremental improvements. It seems like we might be close to maxing out what the current iteration of LLMs can accomplish and we're into the diminishing returns phase. If that's the case, then I have a bad feeling for the state of our industry. My experience with LLMs is that their code does _not_ cut i…

Under what metrics are you judging these improvements? If you're talking about improving benchmark scores, as others have pointed out, those are increasing at a regular rate (putting aside the occasional questionable training practices where the benchmark is in the training set). But most individuals seem to be judging "order of magnitude jumps" in terms of whether the model can solve a very specific set of their use cases to a given level of satisfaction or not. This is a highly nonlinear metric, so changes will always appear to be incremental until suddenly it isn't. Judging progress in this way is alchemy, and leads only to hype cycles.

Every indication I've seen is that LLMs are continuing to improve, each fundamental limitation recognized is eventually overcome, and there are no meaningful signs of slowing down. Unlike prior statistical models which have fundamental limitations without solutions, I have not seen evidence to suggest that any particular programming task that can be achieved by humans cannot eventually be solvable by LLM variants. I'm not saying that they necessarily will be, of course, but I'd feel a lot more comfortable seeing evidence that they won't.

Re: Claude 4

#776

Post by viraptor »

elcritch wrote: post #768
icpmacdo wrote: post #754

Earlier quoted context omitted.

"It feels like these new models are no longer making order of magnitude jumps, but are instead into the long tail of incremental improvements. It seems like we might be close to maxing out what the current iteration of LLMs can accomplish and we're into the diminishing returns phase." SWE bench from ~30-40% to ~70-80% this year

Yet despite this all the LLMS I've tried struggle to scale beyond much more than a single module. They're vast improvements on that test perhaps, but in real life they still struggle to be coherent over larger projects and scales.

Those are different kind of issues. Improving the quality of actions is what we're seeing here. Then for the larger projects/contexts the leaders will have to battle it out between the improved agents, or actually moving to something like RWKV and processing the whole project in one go.

Re: Claude 4

#777

Post by tzury »

lxgr wrote: post #206
minimaxir wrote: post #118

An important note not mentioned in this announcement is that Claude 4's training cutoff date is March 2025, which is the latest of any recent model. (Gemini 2.5 has a cutoff of January 2025) https://docs.anthropic.com/en/docs/about-claude/models/overv...

With web search being available in all major user-facing LLM products now (and I believe in some APIs as well, sometimes unintentionally), I feel like the exact month of cutoff is becoming less and less relevant, at least in my personal experience. The models I'm regularly using are usually smart enough to figure out that they should be pulling in new information for a given topic.

web search is an immediate limited operation training is a petabytes long term operation

Re: Claude 4

#778

Post by viraptor »

avs733 wrote: post #769
icpmacdo wrote: post #754

Earlier quoted context omitted.

"It feels like these new models are no longer making order of magnitude jumps, but are instead into the long tail of incremental improvements. It seems like we might be close to maxing out what the current iteration of LLMs can accomplish and we're into the diminishing returns phase." SWE bench from ~30-40% to ~70-80% this year

3% to 40% is a 13x improvement 40% to 80% is a 2x improvement It’s not that the second leap isn’t impressive, it just doesn’t change your perspective on reality in the same way.

Maybe... It will be interesting to see the improvements now compared to other benchmarks. Is 80->90% going to be an incremental fix with minimal impact on the next benchmark (same work but better), or is it going to be an overall 2x improvement on the remaining unsolved cases. (different approach tackling previously missed areas)

It really depends on how that remaining improvement happens. We'll see it soon though - every benchmark nearing 90% is being replaced with something new. SWE-verified is almost dead now.

Re: Claude 4

#779

Post by morsecodist »

dbingham wrote: post #743

It feels like these new models are no longer making order of magnitude jumps, but are instead into the long tail of incremental improvements. It seems like we might be close to maxing out what the current iteration of LLMs can accomplish and we're into the diminishing returns phase. If that's the case, then I have a bad feeling for the state of our industry. My experience with LLMs is that their code does _not_ cut i…

I agree on the diminishing returns and that the code doesn't cut it on its own. I really haven't noticed a significant shift in quality in a while. I disagree on the productivity though.

Even for something like a script to do some quick debugging or answering a question it's been a huge boon to my productivity. It's made me more ambitious and take on projects I wouldn't have otherwise.

I also don't really believe that workers are currently being replaced by LLMs. I have yet to see a system that comes anywhere close to replacing a worker. I think these layoffs are part of a trend that started before the LLM hype and it's just a convenient narrative. I'm not saying that there will be no job loss as a result of LLMs I'm just not convinced it's happening now.

Re: Claude 4

#780

Post by andyferris »

avs733 wrote: post #769
icpmacdo wrote: post #754

Earlier quoted context omitted.

"It feels like these new models are no longer making order of magnitude jumps, but are instead into the long tail of incremental improvements. It seems like we might be close to maxing out what the current iteration of LLMs can accomplish and we're into the diminishing returns phase." SWE bench from ~30-40% to ~70-80% this year

3% to 40% is a 13x improvement 40% to 80% is a 2x improvement It’s not that the second leap isn’t impressive, it just doesn’t change your perspective on reality in the same way.

I wouldn’t want to wait ages for Claude Code to fail 60% of the time.

A 20% risk seems more manageable, and the improvements speak to better code and problem solving skills around.

Post reply on HN