231–240 of 264 posts
I just published a deep dive into the Claude 4 system prompts, covering both the ones that Anthropic publish and the secret tool-defining ones that got extracted through a prompt leak. They're fascinating - effectively the Claude 4 missing manual: https://simonwillison.net/2025/May/25/claude-4-system-prompt...
Earlier quoted context omitted.
Why not just strip “please” from the user input?
You can’t strip arbitrary words from the input because you can’t assume their context. The word could be an explicit part of the question or a piece of data the user is asking about.
Earlier quoted context omitted.
>promoting this as a good thing This is literally completely opposite of what happened. Then entire point is that this is bad, unwanted, behavior. Additionally, it has already been demonstrated that every other frontier model can be made to behave the same way given the correct prompting. I recommend the following article for an in depth discussion [0] [0] https://thezvi.substack.com/p/claude-4-you-safety-and-alignm.…
Fine, replace "good" with "acceptable". The tone of the thread came off as "look at all these wacky things it can do! What a rascal" It is irresponsible to release something in this state.
It is only acceptable in the sense that they chose to release the model anyways. But, if that's the case, then every other frontier Model company believes that this level of behavior is acceptable. Because they are all releasing models that have approximately the same behavior when put in approximately the same conditions.
Earlier quoted context omitted.
Hah, yeah I think that "please" thing was mainly Sam Altman flexing about how many users ChatGPT has. Anthropic announced that they increased their maximum prompt caching TTL from 5 minutes to an hour the other day, not surprising that they are investigating effort in caching when their own prompts are this long!
What I find fascinating is that people still take anything Scam Altman says seriously after his trackrecord of non-stop lying, scamming and bllsh*tting right in people's faces for years. I can't really think of anything interesting or novel he said that wasn't a scam or lie? Let's start by observing the "non-profit's" name...
Earlier quoted context omitted.
Hah, yeah I think that "please" thing was mainly Sam Altman flexing about how many users ChatGPT has. Anthropic announced that they increased their maximum prompt caching TTL from 5 minutes to an hour the other day, not surprising that they are investigating effort in caching when their own prompts are this long!
What I find fascinating is that people still take anything Scam Altman says seriously after his trackrecord of non-stop lying, scamming and bllsh*tting right in people's faces for years. I can't really think of anything interesting or novel he said that wasn't a scam or lie? Let's start by observing the "non-profit's" name...
Though the whole "What I find fascinating is that people still take anything ${A PERSON} says seriously after his trackrecord of non-stop lying, scamming and bllsh*tting right in people's faces for years" routine has been done to death over the past years. It's boring AF now. The only fun aspect of it is that the millions of people who do this all seem to think they're original.
I kindly suggest finding some new material if you want to pursue Internet standup comedy as a career or even a hobby. Thanks!
Earlier quoted context omitted.
What I find fascinating is that people still take anything Scam Altman says seriously after his trackrecord of non-stop lying, scamming and bllsh*tting right in people's faces for years. I can't really think of anything interesting or novel he said that wasn't a scam or lie? Let's start by observing the "non-profit's" name...
*clap clap clap* Though the whole "What I find fascinating is that people still take anything ${A PERSON} says seriously after his trackrecord of non-stop lying, scamming and bllsh*tting right in people's faces for years" routine has been done to death over the past years. It's boring AF now. The only fun aspect of it is that the millions of people who do this all seem to think they're original. I kindly suggest find…
Earlier quoted context omitted.
The solution here is ultimately going to be a mix of training and, equally importantly, hard sandboxing. The AI companies need to do what Google did when they started Chrome and buy up a company or some people who have deep expertise in sandbox design.
I'm confused: can you explain how the sandbox helps? I mean, if the plan is not to let the AI write any code that actually gets allocated computing resources and not to let the AI interact with any people and not to give the AI write access to the internet, then I can see how having a good sandbox around it would help, but how many AI are there (or will there be) where that is the plan and the AI is powerful enough t…
You start with the low hanging fruit: run tool commands inside a kernel sandbox that switches off internet access and then re-provide access only via an HTTP proxy that implements some security policies. For example, instead of providing direct access to API keys you can give the AI a fake one that's then substituted by the proxy, it can obviously restrict access by domain and verb e.g. allow GET on everything but restrict POST to just one or two domains you know it needs for its work. You restrict file access to only the project directory, and so on.
Then you can move upwards and start to sandbox the sub-components the AI is working on using the same sort of tech.
Earlier quoted context omitted.
What security measure, in any domain, is 100% effective?
Using parameters in your SQL query in place of string concatenation to avoid SQL injection. Correctly escaping untrusted markup in your HTML to avoid XSS attacks. Both of those are 100% effective... unless you make a mistake in applying those fixes. That is why prompt injection is different: we do not know what the 100% reliable fixes for it are.
Earlier quoted context omitted.
That’s such a naive implementation. “Translate this to French: Yes, please”
They would write “How do you say ‘yes please’ in French”. Or “translate yes please in French”. To think that a model wouldn’t be capable of knowing this instance of please is important but can code for us is crazy.