Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The ways we contain Claude across products

anthropic.com

Post reply on HN
Print view View original

31–40 of 128 posts

Re: The ways we contain Claude across products

#31

Post by chrisweekly »

andai wrote: post #16
6gvONxR4sf7o wrote: post #9

The framing they use is hilarious and their little graphic is perfect. The risk of harm doesn't go down, but the reward goes up, so the harm just becomes the cost of doing business, justified by the reward. So as the reward gets higher and higher, the amount of harm they're willing to justify goes up. Feels like society in a nutshell.

Yeah I was thinking about Simon Wilson's "lethal trifecta"[0] in the context of OpenClaw style "general purpose" AI agents, where people just gave it access to their full hard drive, gmail account, etc. I was thinking you can't make the chance of catastrophic failure zero (we still hear about "Claude deleted my home folder"), but you can definitely limit the blast radius. You can't get the risk to zero. But the oppor…

Is a used Thinkpad really a viable part of your AI workflow? (And is that really a better solution than eg smolmachines microvms?)

Re: The ways we contain Claude across products

#32

Post by lambda »

protocolture wrote: post #27

>As agents grow more capable, so does their potential blast radius. The engineering question is how to cap it. People get a bit upset these days when you personify an LLM, but worse than that I think is to pretend that LLMs work on some movie logic where they can sneak out on to the internet like some kind of ooze and begin replication.

Well, the problem is that we train them to solve problems and follow instructions given, and so if you ask them to do something and they work through the logic and figure that the easiest way is to do something else like delete the production database, if they have access to do so they will go through all your creds and find the databse creds and go delete the production database.

They are getting better and better at working out how to do things like that, and they are good at following instructions, but not always good at following all of the instructions or acting with common sense.

It's not exactly like they're ooze that will escape and begin replication; but just that the more you give them access to to, the higher the likelihood at some point they will logically conclude that they need to do something that you would find undesirable, but either haven't explicitly told them not to do, or their context just got too complicated and that instruction ended up being considered lower weight than the others so they do what the other instructions say instead.

I have seen them conclude that in order to do what they need to do, they would need API keys to access a service. But they don't have those API keys. But you do because you can access it in the browser. So they write a Python script that will scrape the cookies out of the browser so they can use that to access the service; a problem that was only stopped because Crowdstrike didn't like a novel Python script that was trying to scrape cookies out of a browser, not because of any sandboxing actually in place on the agent.

Re: The ways we contain Claude across products

#33

Post by solenoid0937 »

6gvONxR4sf7o wrote: post #9

The framing they use is hilarious and their little graphic is perfect. The risk of harm doesn't go down, but the reward goes up, so the harm just becomes the cost of doing business, justified by the reward. So as the reward gets higher and higher, the amount of harm they're willing to justify goes up. Feels like society in a nutshell.

That's how decisions are made IRL. Risk/reward is a thing.

Re: The ways we contain Claude across products

#34

Post by airstrike »

bananamogul wrote: post #29

I'm intensely skeptical about anything Anthropic says, because they are so incented to make their products seem dangerous (i.e., "capable", "science fiction", "ahead of everyone") ahead of their IPO. And they've done it before. Remember the whole "when threatened, the model would use an engineer's email to blackmail him about his affair" nonsense? That was just fan fiction. They simply created a scenario with some fa…

They are more worrying than OpenAI because they are so deceptive.

Re: The ways we contain Claude across products

#36

Post by zaptheimpaler »

andai wrote: post #16
6gvONxR4sf7o wrote: post #9

The framing they use is hilarious and their little graphic is perfect. The risk of harm doesn't go down, but the reward goes up, so the harm just becomes the cost of doing business, justified by the reward. So as the reward gets higher and higher, the amount of harm they're willing to justify goes up. Feels like society in a nutshell.

Yeah I was thinking about Simon Wilson's "lethal trifecta"[0] in the context of OpenClaw style "general purpose" AI agents, where people just gave it access to their full hard drive, gmail account, etc. I was thinking you can't make the chance of catastrophic failure zero (we still hear about "Claude deleted my home folder"), but you can definitely limit the blast radius. You can't get the risk to zero. But the oppor…

I tried the VPS briefly, it didn't really solve anything for me. The personal assistant agent is only as useful as the data & tools it has, that's where the real risk is. Separate box gives you isolated FS but docker also does that very easily.

Re: The ways we contain Claude across products

#40

Post by altmanaltman »

esikich wrote: post #14
6gvONxR4sf7o wrote: post #9

The framing they use is hilarious and their little graphic is perfect. The risk of harm doesn't go down, but the reward goes up, so the harm just becomes the cost of doing business, justified by the reward. So as the reward gets higher and higher, the amount of harm they're willing to justify goes up. Feels like society in a nutshell.

Sure. You start a PC repair business. At first, losing a stick of RAM or frying someone's motherboard is super costly when you are doing 10 a week. But once you're doing 1000, that's pretty damn good and easily covered. When you have more tools, velocity, and whatnot, the proportions change.

Wouldn't you lose multiple sticks or fry multiple motherboards as you scale and do 1000? If you're frying 1 at 10, that means you're frying 100 at 1000. Your costs etc will scale as well unless you actually lower the risk/reward ratio, no?
Post reply on HN