Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Claude Code 2.0

npmjs.com

Post reply on HN
Print view View original

331–340 of 431 posts

Re: Claude Code 2.0

#331

Post by simonw »

saberience wrote: post #308
simonw wrote: post #49

Something I realized about this category of tool (I call them "terminal agents" but that already doesn't work now there's an official VS Code extension for this - maybe just "coding agents" instead) is that they're actually an interesting form of general agent . Claude Code, Codex CLI etc can effectively do anything that a human could do by typing commands into a computer. They're incredibly dangerous to use if you d…

Incredibly dangerous to use? Seems like a wild exaggeration. I’ve been using Claude code since launch, must have used it for 1000 hours or more by now, and it’s never done anything I didn’t want it to do. Why would I run it in a sandbox? It writes code for me and occasionally runs a build and tests. I’m not sure why you’re so fixated on the “danger”, when you use these things all the time you end up realizing that th…

You've been safe since launch because you haven't faced an adversarial prompt injection attack yet.

You (and many, many others) likely won't take this threat seriously until adversarial attacks become common. Right now, outside of security researcher proof of concepts, they're still vanishingly rare.

You ask why I'm obsessed with the danger? That's because I've been tracking prompt injection - and our total failure to find a robust solution for it - for three years now. I coined the name for it!

The only robust solution for it that I trust is effective sandboxing.

Re: Claude Code 2.0

#333

Post by ishouldbework »

wrasee wrote: post #309
numpad0 wrote: post #284

fyi: for chatboxes that may take CJK inputs, you MUST use "shift+enter to send" pattern. There is a reason why most multinational chat/LLM app providers always do that instead of simple enter to send even for single-line chatboxes; because plain enter to send breaks input for CJK users. Specifically, Input Method Editors needed for CJK inputs(esp. for C and J), to convert ambiguous semi-readable forms into proper rea…

What’s CJK input? I’m guessing Chinese Japanese Korean?

[dead]

Re: Claude Code 2.0

#334

Post by simonw »

saberience wrote: post #320
guhcampos wrote: post #315

Earlier quoted context omitted.

It is dangerous. Just yesterday my cursor agent made some changes to a live kubernetes cluster even over my specific instruction not to. I gave it kubectl to analyze and find the issues with a large Prometheud + AlertManager configuration, then switched windows to work on something else. When I was back the MF was patching live resources to try and diagnose the issue.

But this is just like giving a junior engineer access to a prod K8s cluster and having them work for hours on stuff related to said cluster... you wouldn't do it. Or at least, I wouldn't do it. In my own career, when I was a junior, I fucked up a prod database... which is why we generally don't give junior/associate people to much access to critical infra. Junior Engineers aren't "dangerous" but we just don't give th…

You and I are advocating for the same exact solution here! Don't give your LLM over-privileged access to production systems.

My way of explaining that to people is to say that it's dangerous to do things like that.

Re: Claude Code 2.0

#336

Post by rglover »

guhcampos wrote: post #315
saberience wrote: post #308

Earlier quoted context omitted.

Incredibly dangerous to use? Seems like a wild exaggeration. I’ve been using Claude code since launch, must have used it for 1000 hours or more by now, and it’s never done anything I didn’t want it to do. Why would I run it in a sandbox? It writes code for me and occasionally runs a build and tests. I’m not sure why you’re so fixated on the “danger”, when you use these things all the time you end up realizing that th…

It is dangerous. Just yesterday my cursor agent made some changes to a live kubernetes cluster even over my specific instruction not to. I gave it kubectl to analyze and find the issues with a large Prometheud + AlertManager configuration, then switched windows to work on something else. When I was back the MF was patching live resources to try and diagnose the issue.

Best way to avoid this is to force the LLM to use git branches for new work. Worst case scenario you lose some cash on tokens and have to toss the branch but your prod system is left unscathed.

Re: Claude Code 2.0

#337

Post by simonw »

Dilettante_ wrote: post #325
brookst wrote: post #318

Earlier quoted context omitted.

That’s a pretty wild misrepresentation. The actual statement was from red team testing in a very contrived and intentional setup designed to test refusal in extreme circumstances. Yes, it was a legit safety issue and worth being aware of, but it’s not it was a general case. Red teamers worked hard to produce that result.

>The actual statement was from red team testing in a very contrived and intentional setup Was it a paper or something? Would you happen to remember the reference?

It's the Claude 4 system card. I wrote about that one here: https://simonwillison.net/2025/may/25/claude-4-system-card/

Re: Claude Code 2.0

#338

Post by simonw »

brookst wrote: post #317
simonw wrote: post #49

Something I realized about this category of tool (I call them "terminal agents" but that already doesn't work now there's an official VS Code extension for this - maybe just "coding agents" instead) is that they're actually an interesting form of general agent . Claude Code, Codex CLI etc can effectively do anything that a human could do by typing commands into a computer. They're incredibly dangerous to use if you d…

Dangerous how? Claude code literally asks before running any command. I suppose they’re dangerous in the same way any terminal shell is dangerous, but it seems a bit of a moral panic. All tools can be dangerous if misused.

Many people (myself included) run them in YOLO mode with approvals turned off, because it's massively more productive. And that's despite me understanding how unsafe that is more than most!

Even with approvals humans will fall victim to dialog fatigue, where they'll click approve on everything without reading it too closely.

Re: Claude Code 2.0

#339

Post by _heimdall »

polyrand wrote: post #260
simonw wrote: post #49

Something I realized about this category of tool (I call them "terminal agents" but that already doesn't work now there's an official VS Code extension for this - maybe just "coding agents" instead) is that they're actually an interesting form of general agent . Claude Code, Codex CLI etc can effectively do anything that a human could do by typing commands into a computer. They're incredibly dangerous to use if you d…

Instead of containers, which may not always be available, I'm experimenting with having control over the shell to whitelist the commands that the LLM can run [0]. Similar to an allow list, but configured outside the terminal agent. Also trying to make it easy to use the same technique in macOS and Linux [0]: https://ricardoanderegg.com/posts/control-shell-permissions-...

This is how I've been using Gemini CLI. It has no permissions by default, whether it wants to search google, run tests, or update a markdown file it has to propose exactly what it needs to do next and I approve it. Often its helpful even just to redirect the LLM, if it starts going down the wrong path I catch it early rather than 20 steps down that road.

I have no way of really guaranteeing that it will do exactly what it proposed and nothing more, but so far I haven't seen it deviate from a command I approved.

Re: Claude Code 2.0

#340

Post by atonse »

stared wrote: post #332

Out of all changes I want the most is to not need to type `\ ` to make a line break.

Early on in claude, I feel like it installed some terminal thing that allowed me to do Shift+Enter directly in the prompt, but I don't remember if that was CC that did it.

So I've been able to shift enter. I'm using iTerm2 and zsh with CC (if that's relevant)

Post reply on HN