Superintelligence: The Idea That Eats Smart People (2016)
1–10 of 253 posts
Re: Superintelligence: The Idea That Eats Smart People (2016)
#2> If you encountered a cheetah in pre-industrial times (and survived the meeting), you might think it was impossible for anything to go faster.
Fun fact, there is no historical evidence of an adult human ever dying from a cheetah attack. They are naturally shy, and a lot smaller than you may realize.
Re: Superintelligence: The Idea That Eats Smart People (2016)
#3>Sam Altman, the man who runs YCombinator, is my favorite example of this archetype. He seems entranced by the idea of reinventing the world from scratch, maximizing impact and personal productivity. He has assigned teams to work on reinventing cities, and is doing secret behind-the-scenes political work to swing the election.
>Such skull-and-dagger behavior by the tech elite is going to provoke a backlash by non-technical people who don't like to be manipulated. You can't tug on the levers of power indefinitely before it starts to annoy other people in your democratic society.
How right the author was.
Re: Superintelligence: The Idea That Eats Smart People (2016)
#4The main problem of the hard takeoff theory is not the abstract nature of the scenario but rather the fact that it makes the same mistake as the unconstrained optimization paradigm, it takes intelligence to be an unconstrained optimization process.
In fact, if we consider the strongest version of the safety argument for AI, namely one in which the danger is not coming from robots but rather from a disembodied AI controlling our global finances and/or infrastructure, the assumption still does not correspond to reality.
Re: Superintelligence: The Idea That Eats Smart People (2016)
#5Some of the themes remind me of themes mentioned in this matrix analysis. Specifically I am reminded of the Dune concept of control: "you control what you can destroy" and then asking "do you control your refrigerator?". Sure, you can turn it off but then your food would rot and you might starve. So in a real sense humans have not controlled machines for a long time but have been co evolving in symbiosis. Sure, it's not driven by natural selection and standard rules of life, but it is important to frame our relationship with machines in new ways if we're ever going to make some sort of artificial intelligence.
Re: Superintelligence: The Idea That Eats Smart People (2016)
#6AI Superintelligence doesn't scare me for the same reasons "grey goo" doesn't scare me.
We are awash in self-replicating machines. The biosphere is already a grey-goo apocalypse. Any new competitors have a serious moat to cross to out compete any existing self-replicators.
We are awash in intelligent agents. Our society (and meta society) is full of superhuman agents already. There is a huge moat for any new intelligence paradigm to cross.
What I am afraid of is the existing superhuman agents (companies, governments and religons) will produce AGI or superintelligence and then proceed to use it as cognitive mitocondria, even further deepening thier supremacy in the cognitive ecosystem.
Re: Superintelligence: The Idea That Eats Smart People (2016)
#7The main problem of the hard takeoff theory is not the abstract nature of the scenario but rather the fact that it makes the same mistake as the unconstrained optimization paradigm, it takes intelligence to be an unconstrained optimization process. In fact, if we consider the strongest version of the safety argument for AI, namely one in which the danger is not coming from robots but rather from a disembodied AI cont…
If anything the hard takeoff theory is too conservative. It turns out you don't need self-improvement to get to superintelligence. You just need a ridiculous amount of money. Where can you get a ridiculous amount of money? The market will give it to you because FOMO.
AI is easier than people 10 years ago thought it would be. It's also easier to align than people feared it would be. It's the humans using the AI that are hard to control.
Re: Superintelligence: The Idea That Eats Smart People (2016)
#8In the big 2026, everything certain people worried about with superintelligence came to fruition and they were vindicated. The people closest to ASI are indicating recursive self improvement is imminent, the smartest engineers in the labs themselves are autonomously using agents to develop and improve the models. The arms race is evident. NVDA is the world's most valuable company determined by the worlds' collective wisdom of those with skin-in-the-game.
If there exists a path of runaway superintelligence, the trajectory we've experienced has been following it to a tee. Their predictive power was affirmed.
All the "AI is a nothingburger" predictions of the last decade, including many here even in the last year, have aged incredibly poorly.
Re: Superintelligence: The Idea That Eats Smart People (2016)
#9It's amusing to read people in the past writing about the prospect of superhuman intelligence. The real problems have turned out to be different. Sycophancy and hallucinations, which are part of being confidently wrong, remains a big problem. Needing square miles of data centers was an issue in 1950s science fiction, and disappeared by the 1980s. Yet now they're being built, with private funding and the prospect of profit. The need for way too much training data indicates something is still wrong with the current approach.
None of that was predicted.
Re: Superintelligence: The Idea That Eats Smart People (2016)
#10Some of the themes remind me of themes mentioned in this matrix analysis. Specifically I am reminded of the Dune concept of control: "you control what you can destroy" and then asking "do you control your refrigerator?". Sure, you can turn it off but then your food would rot and you might starve. So in a real sense humans have not controlled machines for a long time but have been co evolving in symbiosis. Sure, it's…
Could you elaborate on that last part?