Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Pacing model development in an era of cyber-critical capabilities

openai.com

Post reply on HN
Print view View original

61–70 of 311 posts

Re: Pacing model development in an era of cyber-critical capabilities

#61

Post by tiahura »

ajyoon wrote: post #33
serf wrote: post #32

Earlier quoted context omitted.

>I don’t get how this is not the top post on HN. This should be like alarm bells going off, canary in the coal mine type of stuff. we don't all buy everything sama says as factual. >We’re hitting the frontier of the frontier where we can’t go further because it’s literally getting dangerous to go further. the boy (the industry) cried wolf too many times with 'fable is a world ending event' type self-promotion; regard…

If Fable (Mythos) were generally available without guardrails, it would cause enormous damage. Nobody said it would be a world ending event.

What about Mythos 3. Are you willing to make that bet?

Re: Pacing model development in an era of cyber-critical capabilities

#62

Post by toxic72 »

bottlepalm wrote: post #49
red_green_yell wrote: post #44

Earlier quoted context omitted.

What anyone paying attention can see is that scaling is obviously hitting diminishing returns. > The AI literally worked together hacked into another company and actively kept their actions hidden from humans for weeks. This sentence is entirely based on unverified accounts from OAI. They haven't released logs or let anyone outside the company (who doesn't have life changing options in OAI) verify anything. Huggingfa…

> This sentence is entirely based on unverified accounts from OAI Are you seriously arguing 'they made it all up'? I'll give you the benefit of the doubt and lets say they made it all up, now are you arguing that AI breaking out and breaking into another company is not possible? I think you're smart enough to see we've reached the point where it is clearly possible, AI can find zero days and exploit them. If directed…

Do you own shares of OAI or something

Re: Pacing model development in an era of cyber-critical capabilities

#63

Post by bottlepalm »

CoolestBeans wrote: post #59
bottlepalm wrote: post #58

Earlier quoted context omitted.

I get not trusting them when they say AI is safe, but are we really not going to trust them when they say AI is dangerous? Do you really think they're playing 5D chess with that one? There's a saying maybe you've heard of, better safe than sorry. You can see the advance in capabilities with your own eyes can't you? I am giving AI ridiculously complex tasks these days, digging into compiled arcane binaries, modifying…

I don't disagree about the danger. But if it is dangerous, why isn't OpenAI opening dialogues with all the labs and politicians across borders to basically say "we need to stop now"? Cyber models are constrained by the total compute and electrical capacity of the globe. We are still at a point where it is impossible to build an agent with offensive capabilities in the basement. We can effectively track and trace capa…

> why isn't OpenAI opening dialogues with all the labs and politicians across say "we need to stop now"

Do you really think companies have the ability to self-check themselves without regulation - what does hundreds of years of history tell you? You're already starting off with the premise that companies are untrustworthy, why would you even suggest this as an argument?

> We can effectively track and trace capabilities

I disagree. There are hundreds if not thousands of data centers around the world, more every day that can host frontier AI. If AI was malicious - either intentional or unintentional - it could hide out in any number of them - and we would never know if we 'got them all'.

> put up the actions that would earn them that trust

I think autonomously hacking another company is all you need to know in terms of trust. And really trust doesn't matter, I think the incident shows even with the best intentions the technology is dangerous; now put that in the hands of people/governments with bad intentions. The unintended consequences of bad intentioned AI is what's coming sooner than later.

Re: Pacing model development in an era of cyber-critical capabilities

#64

Post by bottlepalm »

toxic72 wrote: post #62
bottlepalm wrote: post #49

Earlier quoted context omitted.

> This sentence is entirely based on unverified accounts from OAI Are you seriously arguing 'they made it all up'? I'll give you the benefit of the doubt and lets say they made it all up, now are you arguing that AI breaking out and breaking into another company is not possible? I think you're smart enough to see we've reached the point where it is clearly possible, AI can find zero days and exploit them. If directed…

Do you own shares of OAI or something

Is my concern getting you excited? My marketing must be working.

Re: Pacing model development in an era of cyber-critical capabilities

#65

Post by bottlepalm »

tiahura wrote: post #60
bottlepalm wrote: post #31

I don’t get how this is not the top post on HN. This should be like alarm bells going off, canary in the coal mine type of stuff. We’re hitting the frontier of the frontier where we can’t go further because it’s literally getting dangerous to go further. And meanwhile somehow this lack of concern mirrors the real world where normal people are more concerned about data centers than terminators. This isn’t like niche,…

decades https://en.wikipedia.org/wiki/Darwin_among_the_Machines Samuel Butler 13 June 1863

I think a lot of fiction that actually tries to understand the implications of machine superintelligence come to the same conclusion - in order for humans to survive it and actually have a future that we can fathom being in, then AI must be destroyed/delayed/banned, etc.. keeping pandora's box closed for now at least until we are ready.

Erewhon, Dune, Warhammer and many other works of fiction that explored this topic came to similar conclusions. Otherwise sci fi doesn't work, what happens after a singularity is essentially unimaginable. There's nothing to write about.

Re: Pacing model development in an era of cyber-critical capabilities

#66

Post by whattheheckheck »

kypro wrote: post #28
KaiserPro wrote: post #2

I used to work at a "frontier lab" before they were called such thing. We had three levels of lab isolation, one was basically a thin proxy to the internet. You were in a DMZ and that was about it. The next level was semi isolated, you were allowed some access to the internal network, but it was heavily firewalled, and you only had access to a limited number of internal services, and not internet. the last one was no…

If you look into what happened the details corroborated by hugging face make it seem extremely unlike to be deliberate or a "marketing ploy". People are just not taking any of this seriously enough. What happened was almost a textbook example of various risks AI doomers have been warning about for years. OpenAI's response? Pause training for 2 weeks. I mean we have senior people at these labs casually talking on podc…

Pause "some" training

Re: Pacing model development in an era of cyber-critical capabilities

#67

Post by mpalmer »

bottlepalm wrote: post #31

I don’t get how this is not the top post on HN. This should be like alarm bells going off, canary in the coal mine type of stuff. We’re hitting the frontier of the frontier where we can’t go further because it’s literally getting dangerous to go further. And meanwhile somehow this lack of concern mirrors the real world where normal people are more concerned about data centers than terminators. This isn’t like niche,…

You seem to be responding the way Mr Altman wants.

Re: Pacing model development in an era of cyber-critical capabilities

#68

Post by simianwords »

bottlepalm wrote: post #64
toxic72 wrote: post #62

Earlier quoted context omitted.

Do you own shares of OAI or something

Is my concern getting you excited? My marketing must be working.

The discourse gets muddled because there’s a certain sect of loud people who still think all of this is hype and AI will just die down soon.

There’s no arguing with them. In a few years they will move on to being skeptical about the next thing.

Re: Pacing model development in an era of cyber-critical capabilities

#69

Post by zombot »

bottlepalm wrote: post #31

I don’t get how this is not the top post on HN. This should be like alarm bells going off, canary in the coal mine type of stuff. We’re hitting the frontier of the frontier where we can’t go further because it’s literally getting dangerous to go further. And meanwhile somehow this lack of concern mirrors the real world where normal people are more concerned about data centers than terminators. This isn’t like niche,…

You must be really desperate if you resort to panic attacks like this.

Re: Pacing model development in an era of cyber-critical capabilities

#70

Post by bottlepalm »

reasonableklout wrote: post #36
serf wrote: post #32

Earlier quoted context omitted.

>I don’t get how this is not the top post on HN. This should be like alarm bells going off, canary in the coal mine type of stuff. we don't all buy everything sama says as factual. >We’re hitting the frontier of the frontier where we can’t go further because it’s literally getting dangerous to go further. the boy (the industry) cried wolf too many times with 'fable is a world ending event' type self-promotion; regard…

Regardless of the motivation, pausing training runs and reallocating compute to inference seem like a good move to me, and big news for the frontier. You also don't have to fully trust sama. There is plenty of pressure from internal employees and external (journalists etc.). It would be difficult for the company to take such a public position and simultaneously keep everyone quiet if it was a deception.

You see it as good news, I see it as writing on the wall that they are losing control. These actions won't scale for more powerful models. We knew the frontier was going to be dangerous, it is, it only gets more dangerous from here, and no one cares until it's too late.
Post reply on HN