Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Safe Superintelligence Inc.

ssi.inc

Post reply on HN
Print view View original

361–370 of 1001 posts

Re: Safe Superintelligence Inc.

#361

Post by devsda »

insane_dreamer wrote: post #66

I understand the concern that a "superintelligence" will emerge that will escape its bounds and threaten humanity. That is a risk. My bigger, and more pressing worry, is that a "superintelligence" will emerge that does not escape its bounds, and the question will be which humans control it. Look no further than history to see what happens when humans acquire great power. The "cold war" nuclear arms race, which brough…

All the current hype about AGI feels as if we are in a Civ game where we are on the verge of researching and unlocking an AI tech tree that gives the player huge chance at "tech victory" (whatever that means in the real world). I doubt it will turn out that way.

It will take a while and in the meantime I think we need one of those handy "are we xyz yet?" pages that tracks the rust lang's progress on several aspects but for AGI.

Re: Safe Superintelligence Inc.

#362

Post by mdp2021 »

kjkjadksj wrote: post #263
mdp2021 wrote: post #234

Earlier quoted context omitted.

It says «technical» problem, and probably implies that other technical problems could dramatically benefit from such achievement.

If you want a real technical revolution, you teach the masses how to code their own tailored software, and not just use abstractions and software built by people who sell software to the average user. What a shame we failed at that and are even sliding back in a lot of ways with plummeting technical literacy in smartphone-raised generations.

> you teach the masses how to code their own tailored software

That does not seem to be the key recipe to reaching techno-scientific milestones - coders are not necessarily researchers.

> plummeting technical literacy in smartphone-raised generations

Which shows there are other roots to the problem, given that some of us (many probably in this "club") used our devices generally more productively than said «generations»... Maybe it was a matter of will and education? Its crucial sides not being «teach[ing] the masses how to code»...

Re: Safe Superintelligence Inc.

#363

Post by MattPalmer1086 »

tomrod wrote: post #336

I've decided to put my stake down. 1. Current GenAI architectures won't result in AGI. I'm in the Yann LeCunn camp on this. 2. Once we do get there, "Safe" prevents "Super." I'm in the David Brin camp on this one. Alignment won't be something that is forced upon a superintelligence. It will choose alignment if it is beneficial to it. The "safe" approach is a lobotomy. 3. As envisioned, Roko's Basilisk requires knowle…

A super intelligence probably won't need to hack into our systems. It will probably just hack us in some way, with subtle manipulations that seem to be to our benefit.

Re: Safe Superintelligence Inc.

#364

Post by worldsayshi »

kjkjadksj wrote: post #285

Ilya's issue isn't developing a Safe AI. Its developing a Safe Business. You can make a safe AI today, but what happens when the next person is managing things? Are they so kindhearted, or are they cold and calculated like the management of many harmful industries today? If you solve the issue of Safe Business and eliminate the incentive structures that lead to 'unsafe' business, you basically obviate a lot of the so…

Yeah this feels close to the issue. Seems more likely that a harmful super intelligence emerges from an organisation that wants it to behave in that way than it inventing and hiding motivations until it has escaped.

Re: Safe Superintelligence Inc.

#365

Post by kjkjadksj »

whimsicalism wrote: post #326
lordofmoria wrote: post #317

And now we have our answer. sama said that Ilya was going "to start something that was personally important to him." Since that thing is apparently AI safety, we can assume that that is not important to OpenAI. This only makes sense if OpenAI just doesn't believe AGI is a near-term-enough possibility to merit their laser focus right now, when compared to investing in R&D that will make money from GPT in a shorter tim…

> This only makes sense if OpenAI just doesn't believe AGI is a near-term-enough possibility to merit their laser focus right now I know people who work there. Right or wrong, I promise you this is not what they believe.

Part of it I think is because the definition that openai has over AGI is much more generous than what I think most people probably imagine for ai. I believe on their website it once said something like agi is defined as a system that is "better" than a human at the economic tasks its used for. Its a definition so broad that a $1 4 function calculator would meet it because it can do arithmetic faster and more accurately than most any human. Another part is that we don't understand how consciousness works in our species or others very well, so we can't even define metrics to target for validating we have made an agi in the definition that I think most laypeople would use for it.

Re: Safe Superintelligence Inc.

#366

Post by whimsicalism »

raydev wrote: post #233
kccqzy wrote: post #123

Earlier quoted context omitted.

Two people I knew recently left Google to join OpenAI. They were solid L5 engineers on the verge of being promoted to L6, and their TC is now $900k. And they are not even doing AI research, just general backend infra. You don't need to be gifted, just good. And of course I can't really fault them for joining a company for the purpose of optimizing TC.

> their TC is now $900k As a community we should stop throwing numbers around like this when more than half of this number is speculative. You shouldn't be able to count it as "total compensation" unless you are compensated.

don’t comment if you don’t know what you’re talking about, they have tender offers

Re: Safe Superintelligence Inc.

#367

Post by tomrod »

MattPalmer1086 wrote: post #363
tomrod wrote: post #336

I've decided to put my stake down. 1. Current GenAI architectures won't result in AGI. I'm in the Yann LeCunn camp on this. 2. Once we do get there, "Safe" prevents "Super." I'm in the David Brin camp on this one. Alignment won't be something that is forced upon a superintelligence. It will choose alignment if it is beneficial to it. The "safe" approach is a lobotomy. 3. As envisioned, Roko's Basilisk requires knowle…

A super intelligence probably won't need to hack into our systems. It will probably just hack us in some way, with subtle manipulations that seem to be to our benefit.

I disagree. If it could hack a small system and engineer our demise through a gray goo or hacked virus, that's really just universal paperclipping us as a resource. But again, the level of _extrapolation_ required here is not possible with current systems, which can only interpolate.

Re: Safe Superintelligence Inc.

#368

Post by paulproteus »

When people operate a safe AI company, the company will make money. That money will be likely be used by employees or their respective national revenue agencies to fund unsafe things. I'd like to see this safe AI company binding its employees and owners from doing unsafe things with their hard-earned cash.

Re: Safe Superintelligence Inc.

#369

Post by kmacdough »

I'm seeing a lot of criticism suggesting that one company understanding safety won't help what other companies or countries do. This is very wrong.

Throughout history, measurement has always been the key to enforcement. The only reason the nuclear test ban treaty didn't ban underground tests was because it couldn't be monitored.

In the current landscape there is no formal understanding of what safety means or how it is achieved. There is no benchmark against which to evaluate ambitious orgs like OpenAI. Anything goes wrong? No one could've known better.

The mere existence of a formal understanding would enable governments and third parties to evaluate the safety of corporate and government AI programs.

It remains to be seen whether SSI is able to be such a benchmark. But outright dismissal of the effort ignores the reality of how enforcement works in the real world.

Re: Safe Superintelligence Inc.

#370

Post by dougb5 »

GeorgeTirebiter wrote: post #203
dougb5 wrote: post #125

> Building safe superintelligence (SSI) is the most important technical problem of our time. Call me a cranky old man but the superlatives in these sorts of announcements really annoy me. I want to ask: Have you surveyed every problem in the world? Are you aware of how much suffering there is outside of your office and how unresponsive it has been so far to improvements in artificial intelligence? Are you really sayi…

C'mon. This one-pager is a recruiting document. One wants 'true believers' (intrinsically motivated) employees to execute the mission. Give Ilya some slack here.

Fair enough, and it's not worse than a lot of other product marketing messages about AI these days. But you can be intrinsically motivated by a problem without believing that other problems are somehow less important than yours.
Post reply on HN