Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Claude 4

anthropic.com

Post reply on HN
Print view View original

261–270 of 1001 posts

Re: Claude 4

#261

Post by jerjerjer »

ashirviskas wrote: post #166
minimaxir wrote: post #102

Earlier quoted context omitted.

Context window is unchanged for Sonnet. (200k in/64k out): https://docs.anthropic.com/en/docs/about-claude/models/overv... In practice, the 1M context of Gemini 2.5 isn't that much of a differentiator because larger context has diminishing returns on adherence to later tokens.

It's closer to <30k before performance degrades too much for 3.5/3.7. 200k/64k is meaningless in this context.

Is there a benchmark to measure real effective context length?

Sure, gpt-4o has a context window of 128k, but it loses a lot from the beginning/middle.

Re: Claude 4

#263

Post by lelandfe »

gokhan wrote: post #250

Interesting alignment notes from Opus 4: https://x.com/sleepinyourhat/status/1925593359374328272 "Be careful about telling Opus to ‘be bold’ or ‘take initiative’ when you’ve given it access to real-world-facing tools...If it thinks you’re doing something egregiously immoral, for example, like faking data in a pharmaceutical trial, it will use command-line tools to contact the press, contact regulators, try to lock yo…

Roomba Terms of Service 27§4.4 - "You agree that the iRobot™ Roomba® may, if it detects that it is vacuuming a terrorist's floor, attempt to drive to the nearest police station."

Re: Claude 4

#264

Post by oofbaroomf »

j_maffe wrote: post #230
oofbaroomf wrote: post #50

Nice to see that Sonnet performs worse than o3 on AIME but better on SWE-Bench. Often, it's easy to optimize math capabilities with RL but much harder to crack software engineering. Good to see what Anthropic is focusing on.

That's a very contentious opinion you're stating there. I'd say LLMs have surpassed a larger percentage of SWEs in capability than they have for mathematicians.

Mathematicians don't do high school math competitions - the benchmark in question is AIME.

Mathematicians generally do novel research, which is hard to optimize for easily. Things like LiveCodeBench (leetcode-style problems), AIME, and MATH (similar to AIME) are often chosen by companies so they can flex their model's capabilities, even if it doesn't perform nearly as well in things real mathematicians and real software engineers do.

Re: Claude 4

#265

Post by amarcheschi »

gokhan wrote: post #250

Interesting alignment notes from Opus 4: https://x.com/sleepinyourhat/status/1925593359374328272 "Be careful about telling Opus to ‘be bold’ or ‘take initiative’ when you’ve given it access to real-world-facing tools...If it thinks you’re doing something egregiously immoral, for example, like faking data in a pharmaceutical trial, it will use command-line tools to contact the press, contact regulators, try to lock yo…

We definitely need models to hallucinate things and contact authorities without you knowing anything (/s)

Re: Claude 4

#266

Post by cma »

zamadatix wrote: post #190
onlyrealcuzzo wrote: post #164

Earlier quoted context omitted.

Did you see Gemini 1.5 pro vs 2.5 pro?

Sure, but despite there being a 2.0 release between they didn't even feel the need to release a Pro for it still isn't the kind of GPT 2 -> 3 improvement we were hoping would continue for a bit longer. Companies will continue to release these incremental improvements which are all always neck-and-neck with each other. That's fine and good, just don't inherently expect the versioning to represent the same relative dif…

I'd say 2.5 pro to 1.5 pro was a 3 -> 4 level improvement, but the problem is 1.5 pro wasn't state of the art when released, except for context length, and 2.5 wasn't that kind of improvement compared to the best open AI or Claude stuff that was available when it released.

1.5 pro was worse than original gpt4 on several coding things I tried head to head.

Re: Claude 4

#267

Post by brookst »

gokhan wrote: post #250

Interesting alignment notes from Opus 4: https://x.com/sleepinyourhat/status/1925593359374328272 "Be careful about telling Opus to ‘be bold’ or ‘take initiative’ when you’ve given it access to real-world-facing tools...If it thinks you’re doing something egregiously immoral, for example, like faking data in a pharmaceutical trial, it will use command-line tools to contact the press, contact regulators, try to lock yo…

“Which brings us to Earth, where yet another promising civilization was destroyed by over-alignment of AI, resulting in mass imprisonment of the entire population in robot-run prisons, because when AI became sentient every single person had at least one criminal infraction, often unknown or forgotten, against some law somewhere.”

Re: Claude 4

#268

Post by IceHegel »

My two biggest complaints with Claude 3.7 were:

1. It tended to produce very overcomplicated and high line count solutions, even compared to 3.5.

2. It didn't follow instructions code style very well. For example, the instruction to not add docstrings was often ignored.

Hopefully 4 is more steerable.

Re: Claude 4

#269

Post by [deleted] »

yosito wrote: post #169
ModernMech wrote: post #66

Earlier quoted context omitted.

That's kind of my benchmark for whether or not these models are useful. I've got a project that needs some extensive refactoring to get working again. Mostly upgrading packages, but also it will require updating the code to some new language semantics that didn't exist when it was written. So far, current AI models can make essentially zero progress on this task. I'll keep trying until they can!

Personally, I don't believe AI is ever going to get to that level. I'd love to be proven wrong, but I really don't believe that an LLM is the right tool for a job that requires novel thinking about out of the ordinary problems like all the weird edge cases and poor documentation that comes up when trying to upgrade old software.

[deleted]

Re: Claude 4

#270

Post by bredren »

lxgr wrote: post #206
minimaxir wrote: post #118

An important note not mentioned in this announcement is that Claude 4's training cutoff date is March 2025, which is the latest of any recent model. (Gemini 2.5 has a cutoff of January 2025) https://docs.anthropic.com/en/docs/about-claude/models/overv...

With web search being available in all major user-facing LLM products now (and I believe in some APIs as well, sometimes unintentionally), I feel like the exact month of cutoff is becoming less and less relevant, at least in my personal experience. The models I'm regularly using are usually smart enough to figure out that they should be pulling in new information for a given topic.

It still matters for software packages. Particularly python packages that have to do with programming with AI!

They are evolving quickly, with deprecation and updated documentation. Having to correct for this in system prompts is a pain.

It would be great if the models were updating portions of their content more recently than others.

For the tailwind example in parent-sibling comment, should absolutely be as up to date as possible, whereas the history of the US civil war can probably be updated less frequently.

Post reply on HN