based on the naming conventions established by OpenAI and StabilityAI, this may be the most dangerous AI company yet
"Definitely-Won't-Go-Full-Skynet-AI" was another name in consideration.
Safe Superintelligence Inc.
61–70 of 1001 posts
Re: Safe Superintelligence Inc.
#62Glad to see Ilya is back in a position to contribute to advancing AI. I wonder how they are going to manage to pay the kinds of compensation packages that truly gifted AI researchers can make now from other companies that are more commercially oriented. Perhaps they can find people who are ideologically driven and/or are already financially independent. It's also hard to see how they will be able to access enough com…
At the end game, a "non-safe" superinteligence seems easier to create, so like any other technology, some people will create it (even if just because they can't make it safe). And in a world with multiple superintelligent agents, how can the safe ones "win"? It seems like a safe AI is at inherent disadvantage for survival.
Re: Safe Superintelligence Inc.
#63> Our singular focus means no distraction by management overhead or product cycles, and our business model means safety, security, and progress are all insulated from short-term commercial pressures.
well, that's some concrete insight into whatever happened at OpenAI. kinda obvious though in hindsight I guess.
Re: Safe Superintelligence Inc.
#64Would you rather your future overlords to be called "The Safe Company" or "The Open Company"?
Galaxy Brain: TransparentAI
Re: Safe Superintelligence Inc.
#65Glad to see Ilya is back in a position to contribute to advancing AI. I wonder how they are going to manage to pay the kinds of compensation packages that truly gifted AI researchers can make now from other companies that are more commercially oriented. Perhaps they can find people who are ideologically driven and/or are already financially independent. It's also hard to see how they will be able to access enough com…
Academic compensation is different than what you’d find elsewhere on Hacker News. Likewise, academic performance is evaluated differently than what you’d expect as a software engineer. Ultimately, everyone cares about scientific impact so academic compensation relies on name and recognition far more than money. Personally, I care about the performance of the researchers (i.e., their publications), the institution’s larger research program (and their resources), the institution’s commitment to my research (e.g., fellowships and tenure). I want to do science for my entire career so I prioritize longevity rather than a quick buck.
I’ll add, the lack of compute resources was a far worse problem early in the deep learning research boom, but the market has adjusted and most researchers are able to be productive with existing compute infrastructure.
Re: Safe Superintelligence Inc.
#66I understand the concern that a "superintelligence" will emerge that will escape its bounds and threaten humanity. That is a risk.
My bigger, and more pressing worry, is that a "superintelligence" will emerge that does not escape its bounds, and the question will be which humans control it. Look no further than history to see what happens when humans acquire great power. The "cold war" nuclear arms race, which brought the world to the brink of (at least partial) annihilation, is a good recent example.
Quis custodiet ipsos custodes? -- That is my biggest concern.
Update: I'm not as worried about Ilya et al as commercial companies (including formerly "open" OpenAI) discovering AGI.
Re: Safe Superintelligence Inc.
#67I am not on the bleeding edge of this stuff. I wonder though: How could a safe super intelligence out compete an unrestricted one? Assuming another company exists (maybe OpenAI) that is tackling the same goal without spending the cycles on safety, what chance do they have to compete?
It can't. Unfortunately.
People spending so much time thinking about the systems (the models) themselves, not enough about the system that builds the systems. The behaviors of the models will be driven by the competitive dynamics of the economy around them, and yeah, that's a big, big problem.
Re: Safe Superintelligence Inc.
#68“””We are assembling a lean, cracked team of the world’s best engineers and researchers dedicated to focusing on SSI and nothing else.”””
Cracked indeed
Re: Safe Superintelligence Inc.
#69How are they gonna pay for their compute costs to get the frontier? Seems hard to attract enough investment while almost explicitly promising no return.
6-figure free compute credits from every major cloud provider to start
6 figures would pay for a week for what he needs. Maybe less than a week
Re: Safe Superintelligence Inc.
#70based on the naming conventions established by OpenAI and StabilityAI, this may be the most dangerous AI company yet
Ah yes, the AI Brand Law: the meaning of adjectives in your company name will invert within a few years of launch.