Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Claude 4 System Card

simonwillison.net

Post reply on HN
Print view View original

11–20 of 264 posts

Re: Claude 4 System Card

#11

Post by sureglymop »

simonw wrote: post #3
saladtoes wrote: post #2

https://www.lakera.ai/blog/claude-4-sonnet-a-new-standard-fo... These LLMs still fall short on a bunch of pretty simple tasks. Attackers can get Claude 4 to deny legitimate requests easily by manipulating third party data sources for example.

They gave a bullet point in that intro which I disagree with: "The only way to make GenAI applications secure is through vulnerability scanning and guardrail protections." I still don't see guardrails and scanning as effective ways to prevent malicious attackers. They can't get to 100% effective, at which point a sufficiently motivated attacker is going to find a way through. I'm hoping someone implements a version o…

I only half understand CaMeL. Couldn't the prompt injection just happen at the stage where the P-LLM devises the plan for the other LLM such that it creates a different, malicious plan?

Or is it more about the user then having to confirm/verify certain actions and what is essentially a "permission system" for what the LLM can do?

My immediate thought is that that may be circumvented in a way where the user unknowingly thinks they are confirming something safe. Analogous to spam websites that show a fake "Allow Notifications" prompt that is rendered as part of the actual website body. If the P-LLM creates the plan it could make it arbitrarily complex and confusing for the user, allowing something malicious to happen.

Overall it's very good to see research in this area though (also seems very interesting and fun).

Re: Claude 4 System Card

#12

Post by Aeolun »

aabhay wrote: post #5

Given the cited stats here and elsewhere as well as in everyday experience, does anyone else feel that this model isn’t significantly different, at least to justify the full version increment? The one statistic mentioned in this overview where they observed a 67% drop seems like it could easily be reduced simply by editing 3.7’s system prompt. What are folks’ theories on the version increment? Is the architecture sig…

I think they didn’t have anywhere to go after 3.7 but 4. They already did 3.5 and 3.7. People were getting a bit cranky 4 was nowhere to be seen.

I’m fine with a v4 that is marginally better since the price is still the same. 3.7 was already pretty good, so as long as they don’t regress it’s all a win to me.

Re: Claude 4 System Card

#18

Post by kubb »

aabhay wrote: post #7
kubb wrote: post #6

Earlier quoted context omitted.

> to justify the full version increment I feel like a company doesn’t have to justify a version increment. They should justify price increases. If you get hyped and have expectations for a number then I’m comfortable saying that’s on you.

That’s an odd way to defend the decision. “It doesn’t make sense because nothing has to make sense”. Sure, but it would be more interesting if you had any evidence that they decided to simply do away with any logical premise for the 4 moniker.

> nothing has to make sense

It does make sense. The companies are expected to exponentially improve LLMs, and the increasing versions are catering to the enthusiast crowd who just need a number to go up to lose their mind over how all jobs are over and AGI is coming this year.

But there's less and less room to improve LLMs and there are currently no known new scaling vectors (size and reasoning have already been largely exhausted), so the improvement from version to version is decreasing. But I assure you, the people at Anthropic worked their asses off, neglecting their families and sleep and they want to show something for their efforts.

It makes sense, just not the sense that some people want.

Re: Claude 4 System Card

#19

Post by retinaros »

aabhay wrote: post #5

Given the cited stats here and elsewhere as well as in everyday experience, does anyone else feel that this model isn’t significantly different, at least to justify the full version increment? The one statistic mentioned in this overview where they observed a 67% drop seems like it could easily be reduced simply by editing 3.7’s system prompt. What are folks’ theories on the version increment? Is the architecture sig…

the big difference is the capability to think during tool calls. this is what makes openAI o3 lookin like magic

Re: Claude 4 System Card

#20

Post by hakonbogen »

juanre wrote: post #17

This is eerily close to some of the scenarios in Max Tegmark's excellent Life 3.0 [0]. Very much recommended reading. Thank you Simon. 0. https://en.wikipedia.org/wiki/Life_3.0

Yeah thought the same thing. I wonder if he has commented on it?
Post reply on HN