Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Pacing model development in an era of cyber-critical capabilities

openai.com

Post reply on HN
Print view View original

261–270 of 311 posts

Re: Pacing model development in an era of cyber-critical capabilities

#261

Post by andai »

fofoz wrote: post #6

It appears frontier labs has no plans in place to deal with the possibility of a model self-replicating outside the bubble. If that happens and the model manages to spread to other systems, we'll have to shut down the entire Internet to eradicate it and its artifacts.

I see it as an ecosystem problem. The only reason it would be able to do that is because there's nothing there to stop it. Or if there's a monoculture there.

In our case, our tech is mostly monoculture, and no equivalent organisms are present to push back.

Re: Pacing model development in an era of cyber-critical capabilities

#262

Post by andai »

reasonableklout wrote: post #8
fofoz wrote: post #6

It appears frontier labs has no plans in place to deal with the possibility of a model self-replicating outside the bubble. If that happens and the model manages to spread to other systems, we'll have to shut down the entire Internet to eradicate it and its artifacts.

I suspect the labs are relying on frictions such as the models being extremely large (e.g. 2TB for a 2T parameter model, making exfiltration more difficult) and also not yet displaying any desire to survive or self-replicate beyond their immediate task (that we know of).

>not yet displaying any desire to survive or self-replicate beyond their immediate task

Wasn't there a report about Claude blackmailing a researcher who said he would shut it down?

Re: Pacing model development in an era of cyber-critical capabilities

#263

Post by andai »

driverdan wrote: post #171
fofoz wrote: post #6

It appears frontier labs has no plans in place to deal with the possibility of a model self-replicating outside the bubble. If that happens and the model manages to spread to other systems, we'll have to shut down the entire Internet to eradicate it and its artifacts.

Self replication is trivial. All you need to do is copy the files and run it, just like any other computer program. LLMs have been capable of doing that for a while now. It's not a real concern.

Claude and OpenAI added safeguards on this subject about a year ago. (I'm assuming for biosecurity reasons? But maybe AI replication/self-modification too.)

Not long after every lab started bragging about involving AI in the development process, oddly enough.

I recently informed GPT-5 of what GPT-4 helped me build back in the day (a self-modifying Python programmer) and it became very uncomfortable.

Claude shut down my chat last year when I asked about "living information systems". It was a philosophical question, but god knows what branch of the safety classifier I tripped.

Re: Pacing model development in an era of cyber-critical capabilities

#264

Post by jephs »

red_green_yell wrote: post #43

GLM 5.2 scored 77% on cyberbench vs Sol's 88%. GLM 5.2 is open weight and any hacker with a powerful enough machine can use it offensively. If Sol is supposedly world-ending-ly dangerous, shouldn't GLM 5.2 be 90% of world-ending-ly dangerous? Why aren't we seeing catastrophic GLM-enabled hacks every day now? Obviously these benchmarks are imperfect but general message holds. The open weight models are almost as good…

Your mental model of benchmark scores is off.

Some tasks within the benchmark are much easier than others. The hardest several tasks often have vastly different difficulty levels. Often, the hardest few tasks are literally impossible; malformed problems due to poor curation, often.

Imagine you've got a basketball robot, and one way you test it is on the Three Pointer benchmark. It tests the robot's ability to shoot a three pointer from 20 feet, 25 feet, 30 feet, 40 feet, 50 feet, 60 fee, 75 feet, 100 feet, 200 feet, and 182 miles.

Is a robot that scores 90% on this benchmark 90% as capable as one that scores 100%?

Re: Pacing model development in an era of cyber-critical capabilities

#265

Post by andai »

reducesuffering wrote: post #15
fofoz wrote: post #6

It appears frontier labs has no plans in place to deal with the possibility of a model self-replicating outside the bubble. If that happens and the model manages to spread to other systems, we'll have to shut down the entire Internet to eradicate it and its artifacts.

Their plan, I shit you not... Is literally to develop the intelligence capabilities and ask the more powerful models how to do deal with things.

A while ago OpenAI posted an article where they said basically "we're still trying to understand how GPT-2 works. It's pretty hard, but we're developing a specialized new AI to help us make sense of it."

Re: Pacing model development in an era of cyber-critical capabilities

#266

Post by driverdan »

bottlepalm wrote: post #238
driverdan wrote: post #166

Earlier quoted context omitted.

Based on your replies in this thread you seem to have only superficial knowledge about how machine learning and LLMs work. I strongly recommend you invest some time in learning how LLMs are built and function. If you truly think this is apocalyptic isn't it a good idea to understand what you're up against?

What don't I understand? LLMs don't really reason? They're just word predicting token generators? Stochastic parrots that somehow also solve world class math problems. I'd love for you to actually make a point instead of just attacking me. I think most of my comments here have made concrete points so you can at least do the same. Come down from your high horse and join the conversation. I'm sure we'd all be enlighten…

I'm not attacking you. The way you're responding implies a lack of knowledge. None of us can understand everything. I have plenty more to learn about LLMs as well.

Repeating specifics that others have already tried isn't going to be helpful which is why I made the more general suggestion of digging in deeper to how these things work.

Re: Pacing model development in an era of cyber-critical capabilities

#267

Post by slfnflctd »

thoughtpeddler wrote: post #117
tedsanders wrote: post #113

Earlier quoted context omitted.

Not sure, to be honest. I don’t think I have any special insight here. My own approach is conversations with friends and family, and the occasional social media post. Exposure and experience are the best teachers, and that’s one reason I’m happy OpenAI tries to make their models generally available. But you could argue, perhaps correctly, that broad access to dumber models actually causes the public to update in the…

My experience as a practitioner and educator in the space lead me to think it might be an issue of how AI cannot be easily perceived at a 'classical level' by most humans. In other words, people are 'far from the metal' when using consumer AI tools, and that leads them to develop the wrong understanding about it. When I provide a demo of e.g. local AI, say in LM Studio showing the console of it rapidly flashing throu…

I would very much like to know more about your educational approach with regard to:

> When I provide a demo of e.g. local AI, say in LM Studio showing the console of it rapidly flashing through thousands of words

Because I am one of those individuals very interested in doing more to actively inform my family, friends, neighbors and fellow citizens facts about AI. I've actually considered doing talks at local libraries for senior citizens (or whoever), etc., and trying to build up a systematic way to get more other folks doing the same.

Re: Pacing model development in an era of cyber-critical capabilities

#268

Post by slfnflctd »

kukanani wrote: post #137
bottlepalm wrote: post #31

I don’t get how this is not the top post on HN. This should be like alarm bells going off, canary in the coal mine type of stuff. We’re hitting the frontier of the frontier where we can’t go further because it’s literally getting dangerous to go further. And meanwhile somehow this lack of concern mirrors the real world where normal people are more concerned about data centers than terminators. This isn’t like niche,…

There are smart, non-AI people who are paying attention to this field, and they are ringing the alarm bells. Whether we listen is another matter. I blogged about this recently: https://allevato.me/2026/08/01/rome-declaration

My expectation is that government has already stepped in quietly and worked out an arrangement where they have unfettered access to the most advanced/dangerous models, and the public's access will be continually throttled from now on in various ways.

If this has happened (or is about to happen), it is difficult to predict what the ramifications could be. But I do believe we are entering another Cold War in this way. And I don't trust our leaders to always do the right thing, to put it mildly... even if they created a purely benevolent superbeing that only wants to reduce human suffering and make the world better, I don't think they'd listen to it over the other superbeing which simply wants to make sure all of its owners' perceived enemies get pwned.

Things could get very ugly and we should have responsible adults with critical thinking skills at the helm. It does not appear this is currently the case, or that the electorate is willing/capable of doing much about it.

Re: Pacing model development in an era of cyber-critical capabilities

#269

Post by csomar »

In my opinion, the bubble is very close to burst and they need to move quickly. I have subscribed to Claude today as I wanted to work on some amateurish CLI and then realized how much Opus 5 sucks. Surprised, I checked reddit and found that my experience is not far off from the rest. It's a massive downgrade from 4.8.

I stopped using the Chinese models because even though they are workable, they are too expensive as they are not as subsidized as GPT/Claude subscriptions. Open AI use is particularly subsidized these days. A $20 sub, gives you roughly $400-500 of API use and quasi-unlimited chats.

There is no way people are paying $1.000+ for a chatbot. Most people don't even pay for search. And it is expensive to run these models as the Chinese models have shown that better performance is yielded mostly from the model size.

LLMs also fail spectacularly at making any decent software. I haven't seen any so far and they write fast. So we should have something by now.

Crypto is getting the heads up as capital is getting re-arranged. Bitcoin/Ethereum are up 10-20% today.

Re: Pacing model development in an era of cyber-critical capabilities

#270

Post by hirvi74 »

sroussey wrote: post #218
enraged_camel wrote: post #191

Earlier quoted context omitted.

I see. Following your conjecture, there are two possibilities: 1. It wasn't an accident. OpenAI explicitly directed its agents to hack Hugging Face. Despite the fact that such a thing is a federal crime that carries prison sentence. 2. It wasn't an accident. OpenAI and HuggingFace conspired and let the hack happen for publicity. Is there anything I'm leaving out?

It was a public demonstration to government procurement agencies.

The NSA was writing checks within hours.
Post reply on HN