Any of this kinda talk these days is just part of the hype train for the LLM companies tulipomania pyramid scheme.
Afaik no-one that is actually working on AGI is anywhere close atm.
Superintelligence: The Idea That Eats Smart People (2016)
31–40 of 253 posts
Re: Superintelligence: The Idea That Eats Smart People (2016)
#32In the big 2026, everything certain people worried about with superintelligence came to fruition and they were vindicated. The people closest to ASI are indicating recursive self improvement is imminent, the smartest engineers in the labs themselves are autonomously using agents to develop and improve the models. The arms race is evident. NVDA is the world's most valuable company determined by the worlds' collective…
[flagged]
I don't follow that argument at all. Many people who claim they're concerned about the end of the world think that the one sliver of hope we have is that the specific people in charge of modern AI research take existential concerns seriously. Even if you expect that won't be enough to save us, it could hardly help the situation to replace them with other people who are less sympathetic to and probably radicalized against existential concerns.
Re: Superintelligence: The Idea That Eats Smart People (2016)
#33Has anyone played SOMA? Spoiler warning. It explores this idea of, what if there's an AI in charge of ensuring mankind survives at all costs. What would it be willing to do, to keep us alive? Would we even recognize the result as human? It's a horror game and it explores all kinds of fascinating and disturbing scenarios. Simulations of human minds. Artificial worlds. Human minds in robot bodies. Genetically modified…
SOMA was a cognito-hazard for me and my roommate in college; we played it in the dark together while on some sort of mild hallucinogen and when it came time for Simon to find that high-pressure dive suit, we lost our minds (no pun intended). Watching the WAU twist its way through PATHOS II in whatever way worked first is a particularly jarring analogy for what has happened to our own profession. I can't help but think it would be nice for Frictional Games to revisit this topic again soon.
Sidenote: It breaks my heart that all the great underwater-settings in media are hotbeds of horror scenarios. I think Subnautica broke the mold for this, here's to hoping the next generation of aquanauts take to the depths from that series.
Re: Superintelligence: The Idea That Eats Smart People (2016)
#34"Recursive self-improvement" is in the same league as "perpetual motion". What would be a way to recursively self-improve algorithms for matrix multiplication (foundations of machine learning and inference)?
I actually agree. At some point, a RSI system has to interact with real-world, and that imposes serialization constraints. It is harder to know how much that slow-down would be and how much speed-up we will get before that. But a RSI cannot simply be a exponential growth forever.
Re: Superintelligence: The Idea That Eats Smart People (2016)
#35Some previous discussions:
2016 https://news.ycombinator.com/item?id=13240811
Re: Superintelligence: The Idea That Eats Smart People (2016)
#36> The AI is all-powerful and gives you what you ask for, but interprets everything in a super-literal way that you end up regretting. I like imagining similar discourse when a more basic tool was invented: "A hammer is like a genie, it's all powerful, but, when you hit something with it, it interprets that super-literally, and it hits it."
Isn't this a misinterpretation of what everyone in the AI safety space is worried about, though? I think the idea is that having an AI that interprets everything in a super-literal way would probably be catastrophic, but we can't even build that. It would be a nice world-ending problem to have.
Re: Superintelligence: The Idea That Eats Smart People (2016)
#37The main problem of the hard takeoff theory is not the abstract nature of the scenario but rather the fact that it makes the same mistake as the unconstrained optimization paradigm, it takes intelligence to be an unconstrained optimization process. In fact, if we consider the strongest version of the safety argument for AI, namely one in which the danger is not coming from robots but rather from a disembodied AI cont…
If anything the hard takeoff theory is too conservative. It turns out you don't need self-improvement to get to superintelligence. You just need a ridiculous amount of money. Where can you get a ridiculous amount of money? The market will give it to you because FOMO. AI is easier than people 10 years ago thought it would be. It's also easier to align than people feared it would be. It's the humans using the AI that a…
Hard to say if AI with true agency is so ‘easy’. We had a breakthrough with language but not necessarily other things.
Re: Superintelligence: The Idea That Eats Smart People (2016)
#38Earlier quoted context omitted.
There's a story about some Kenyans outrunning a Cheetah in 6km. It had been killing their livestock, so they decided to go after it. Cheetahs are very fast, but humans have way more endurance.
I think the term is persistence hunting ( https://en.wikipedia.org/wiki/Persistence_hunting ). There was an intriguing blog post years ago (which sadly I can’t find) about how terrifying a fictional portrayal of persistence hunting would be- grinding down prey through exhaustion, thinking you’ve escaped but being slowly tracked down.
There's also the story "Go, Go, Go, Said the Bird" by Sonya Dorman in the first Dangerous Visions which is like this, in a way.
Re: Superintelligence: The Idea That Eats Smart People (2016)
#39Earlier quoted context omitted.
There's a story about some Kenyans outrunning a Cheetah in 6km. It had been killing their livestock, so they decided to go after it. Cheetahs are very fast, but humans have way more endurance.
I think the term is persistence hunting ( https://en.wikipedia.org/wiki/Persistence_hunting ). There was an intriguing blog post years ago (which sadly I can’t find) about how terrifying a fictional portrayal of persistence hunting would be- grinding down prey through exhaustion, thinking you’ve escaped but being slowly tracked down.
Somehow that reminds of the old B-movie Surviving the Game with Ice-T.
Re: Superintelligence: The Idea That Eats Smart People (2016)
#40Earlier quoted context omitted.
If anything the hard takeoff theory is too conservative. It turns out you don't need self-improvement to get to superintelligence. You just need a ridiculous amount of money. Where can you get a ridiculous amount of money? The market will give it to you because FOMO. AI is easier than people 10 years ago thought it would be. It's also easier to align than people feared it would be. It's the humans using the AI that a…
Hard to say if AI with true agency is so ‘easy’. We had a breakthrough with language but not necessarily other things.
[dead]