Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Dario, Please

pop.rdi.sh

Post reply on HN
Print view View original

311–320 of 335 posts

Re: Dario, Please

#311

Post by includenotfound »

zahlman wrote: post #179
includenotfound wrote: post #160

Earlier quoted context omitted.

The "sandbox" they used was apparently made of thin paper exposed under a day of heavy rain, too. You'd think, if they truly believed the model is so dangerous, they'd run it in a VM without a network adapter.

I brought this up to someone else and was told that airgapping is apparently much more expensive than I'd naively think. I still think this is a sign that they are not taking their own rhetoric seriously.

ChatGPT paid subscriptions already give you a VM [0]. OpenAI couldn't spare a few VMs for the super ultra mega dangerous evals where they asked the agents to specifically go hack stuff?

[0] https://news.ycombinator.com/item?id=49718530

Re: Dario, Please

#312

Post by pipes »

tanseydavid wrote: post #9
pipes wrote: post #5

Earlier quoted context omitted.

I can't find the quote in the article (and for anyone wondering which song "shook ones", which is an amazing track).

Heading: "OAI-HF Incident" First sentence: "This incident has Dario shook, there ain’t no such thing as halfway crooks."

Thanks. First time searching with my browser return no hits for shook no idea why

Re: Dario, Please

#313

Post by S-E-P »

Conveniently the only companies that warn of a new and dangerous advancement are the ones hemorrhaging money.

Remember when Mythos marked the end of man and the birth of a new race of bots nuking secure systems from orbit?

If a company of a bunch of startup grifters can come up with a tactical nuke of LLM supremacy, why can't the brightest China has to offer already have that technology?

Re: Dario, Please

#314

Post by zahlman »

includenotfound wrote: post #311
zahlman wrote: post #179

Earlier quoted context omitted.

I brought this up to someone else and was told that airgapping is apparently much more expensive than I'd naively think. I still think this is a sign that they are not taking their own rhetoric seriously.

ChatGPT paid subscriptions already give you a VM [0]. OpenAI couldn't spare a few VMs for the super ultra mega dangerous evals where they asked the agents to specifically go hack stuff? [0] https://news.ycombinator.com/item?id=49718530

The issue as I understand it is that they didn't expect Artifactory to be vulnerable in the way it was, or else were completely not paying attention.

But I'm calling for something stronger than a VM here because we shouldn't rely on the VM being bulletproof just like we shouldn't rely on Artifactory being bulletproof. The access should be controlled at the hardware level. Like, networking on internal LANs only, and the entire thing inside a nice big Faraday cage just in case.

Re: Dario, Please

#315

Post by zahlman »

includenotfound wrote: post #310
fernandotakai wrote: post #275

Earlier quoted context omitted.

it's really weird to hear frontier labs say "our internal models are basically AGI" while also saying "airgapping is too hard uwu". if your internal models are so damn good, they should be able to "one shot" airgapping... right?

The best part is, they are totally able to one shot airgapping. You are too. Go ask your local agent to set up a pre-configured Linux VM with whatever stuff you want on it, then ask it to airgap it allowing only X, Y, Z services. It will one shot it. You know what's better? They already do this per (paid) user - your ChatGPT subscription comes with a Linux VM that you can even legitimately SSH into, just ask your age…

Anything that's accomplished simply by running software does not qualify as "airgapping" in my view. The entire premise is that real-world software is buggy and the LLM is much better at locating and exploiting those bugs than you are at preventing them.

But you can only connect to Wifi if you have Wifi hardware, and RF signals are contained by Faraday cages. Ethernet is still a thing for local connections.

Re: Dario, Please

#316

Post by zahlman »

exfalso wrote: post #256
amouat wrote: post #243

Earlier quoted context omitted.

Isn't that exactly what they did? The bots could only access the jfrog instance, so they hacked jfrog?

No that's not what they did, they exposed jfrog raw. It would have been so extremely simple to gate services they need the llm to access... I mean, jfrog was not written with this kind of threat model in mind, and neither were a lot of other tools

> jfrog was not written with this kind of threat model in mind, and neither were a lot of other tools

And we don't just magically know all the consequences of that.

Which is exactly why we do need full, physical air gapping. (Which, yes, would also include self-hosting a mirror of the package repo, if the point of the simulation is to see what's possible with the real package repo.)

Re: Dario, Please

#317

Post by zahlman »

fernandotakai wrote: post #275
zahlman wrote: post #179

Earlier quoted context omitted.

I brought this up to someone else and was told that airgapping is apparently much more expensive than I'd naively think. I still think this is a sign that they are not taking their own rhetoric seriously.

it's really weird to hear frontier labs say "our internal models are basically AGI" while also saying "airgapping is too hard uwu". if your internal models are so damn good, they should be able to "one shot" airgapping... right?

Real air gapping isn't conceptually difficult. It's just a lot of effort and expense (although as far as I can tell, tiny compared to what's involved in building entire new data centres), and adds friction to your operation. (And you also have to have competent, trustworthy employees, as Stuxnet showed us.)

Re: Dario, Please

#318

Post by threecheese »

For persistence, I think the most glaring gap is the inference - if these rogue agents need to call Claude's API, then they aren't persistent - Anthropic can turn them off. A memory resident program needs disk to be persistent, but above all it needs CPU; without it there are only bits.

And so the implication is the swarm has access to enough local compute to perform its own inference, using a large enough model to provide the capabilities to be dangerous.

This is impractical today.

It might be practical tomorrow - if the Chinese are allowed to continue to develop large open weight models that we can quantize and ablate and make small enough to run on a million standard PCs.

Or if *anyone* is allowed to continue to develop AI in the way every other technology has developed - improving, shrinking, optimizing.

And so the argument REALLY isn't against the Chinese - it's that we can never allow this technology to advance outside of trusted labs. If they get their way, they will need to keep this tech locked down forever, which will require much much more than what they are asking.

Re: Dario, Please

#319

Post by zahlman »

w4der wrote: post #282
zahlman wrote: post #179

Earlier quoted context omitted.

I brought this up to someone else and was told that airgapping is apparently much more expensive than I'd naively think. I still think this is a sign that they are not taking their own rhetoric seriously.

> I brought this up to someone else and was told that airgapping is apparently much more expensive than I'd naively think. These labs are one of the most valuable and heavily funded enterprises in the whole world, that they can't properly air-gap their systems to me reads as if their "agents" and LLMs are not as good as they say they are, because if they were, why would it be hard/expensive to air gap a system? They…

I'm not even talking about anything that the agents could help with. I'm talking about physical, real-world measures like https://en.wikipedia.org/wiki/Faraday_cage , removing Wifi hardware and running Ethernet cable for your local server, etc.

Re: Dario, Please

#320

Post by zahlman »

notsylver wrote: post #283
pjc50 wrote: post #281

Earlier quoted context omitted.

Not that airgapping is expensive so much as it's really, really inconvenient once you take it seriously. You need to build special rooms for it, you can't just API out to a datacenter. You need to have processes for requesting data be sent into the box. And so on.

I feel like there is a reasonable compromise between "yeah they have full internet access" and "separate airgapped rooms that require multiple levels of authorization to access" that would make this a lot better without that much more work. I feel like they're doing it intentionally to show how dangerous these models are and that the government must step in and protect them

Compromise is possible; but if the stakes really are what the companies claim they are, and given the amount of money already involved, I really don't see why we should let them compromise.
Post reply on HN