61–70 of 159 posts
> we may have AI that is more capable than everyone in only 1-2 years There's no evidence this will be the case...
I'm starting to think people who build these models are more likely to suffer from AI psychosis.
This is obviously bullshit. If he were really worried about the things he says he is, he'd put the brakes on his company, or would never have started it in the first place.
So if someone (actually, practically everyone) who runs an AI company says AI is dangerous, it's bullshit. If someone who is holding NVDA put options says it, they're talking their book. If someone whose job is threatened by AI says it, it's cope. If someone who doesn't use AI says it, it's fear of change. Is there someone in particular you want to hear it from, or are you completely immune to argument?
My instinct is to take his words as a marketing pitch.
When he says AI is dangerous, it is a roundabout way to say it is powerful and should be taken seriously.
Earlier quoted context omitted.
So if someone (actually, practically everyone) who runs an AI company says AI is dangerous, it's bullshit. If someone who is holding NVDA put options says it, they're talking their book. If someone whose job is threatened by AI says it, it's cope. If someone who doesn't use AI says it, it's fear of change. Is there someone in particular you want to hear it from, or are you completely immune to argument?
> So if someone (actually, practically everyone) who runs an AI company says AI is dangerous, it's bullshit. My instinct is to take his words as a marketing pitch. When he says AI is dangerous, it is a roundabout way to say it is powerful and should be taken seriously.
Earlier quoted context omitted.
Software is more amenable to LLMs because there is a rich source of highly relevant training data that corresponds directly to the building blocks of software, and the "correctness" of software is quasi-self-verifiable. This isn't true for pretty much anything else.
The more verifiable the domain the better suited. We see similar reports of benefits from advanced mathematics research from Terrence Tao, granted some reports seem to amount to very few knew some data existed that was relevant to the proof, but the LLM had it in its training corpus. Still, verifiably correct domains are well-suited. So the concept formal verification is as relevant as ever, and when building interco…
For the easy-to-verify fields like coding, you can train "domain intuitions" directly to the LLM (and some of this training should generalize to other knowledge work abilities), but for other fields you would need to supply them in the prompt as the abilities cannot be trained into the LLM directly. This will need better models but might become doable in a few generations.
The framing of AI risk as a "rite of passage" resonates with me. The "autonomy risks" section is what I think about most. We've seen our agents do unexpected things when given too much latitude. Not dangerous, just wrong in ways we didn't anticipate. The gap between "works in testing" and "works in production" is bigger than most people realize. I'm less worried about the "power seizure" scenario than the economic di…
What makes you think that they will just keep improving? It's not obvious at all, we might soon hit a ceiling, if we haven not already - time will tell. There are lots of technologies that have been 99% done for decades; it might be the same here.
What used to require specialized integration can now be accomplished by a generalized agent.
Earlier quoted context omitted.
What makes you think that they will just keep improving? It's not obvious at all, we might soon hit a ceiling, if we haven not already - time will tell. There are lots of technologies that have been 99% done for decades; it might be the same here.
Which technologies have been 99% "done" for "decades?" Bicycles? carbon fiber frames, electronic shifting, tubeless tires, disc brakes, aerodynamic research Screwdrivers? impact drivers, torque-limiting mechanisms, ergonomic handles Glass? gorilla glass, smart glass, low-e coatings Tires? run-flats, self-sealing, noise reduction Hell even social technologies improve! How is a technology "done?"
It's interesting just how many opinions Amodei shares with AI 2027's authors despite coming from a pretty different context. - Prediction of exponential AI research feedback loops (AI coding speeding up AI R&D) which Amodei says is already starting today - AI being a race between democracies and autocracies with winner-takes-all dynamics, with compute being crucial in this race and global slowdown being infeasible -…
Early "rationalist" community was concerned with AI in this way 20 years ago. Eliezer inspired and introduced the founders of Google DeepMind to Peter Thiel to get their funding. Altman acknowledged how influential Eliezer was by saying how he is most deserving of a Nobel Peace prize when AGI goes well (by lesswrong / "rationalist" discussion prompting OpenAI). Anthropic was a more X-risk concerned fork of OpenAI. Paul Christiano inventor of RLHF was big lesswrong member. AI 2027 is an ex-OpenAI lesswrong contributor and Scott Alexander, a centerpiece of lesswrong / "rationalism". Dario, Anthropic CEO, sister is married to Holden Karnofsky, a centerpiece of effective altruism, itself a branch of lesswrong / "rationalism". The origin of all this was directionally correct, but there was enough power, $, and "it's inevitable" to temporarily blind smart people for long enough.
The EPUB + PDF version is here: https://www.adithyan.io/blog/kindle-ready-adolescence-of-tec...
Earlier quoted context omitted.
Dario and Anthropic's strategy has been to exaggerate the harmful capabilities of LLMs and systems driven by LLMs, positioning Anthropic themselves as the "safest" option. Take from this what you will. As an ordinary human with no investment in the game, I would not expect LLMs to magically work around the well-known physical phenomena that make submarines hard to track. I think there could be some ability to augment…
> Dario and Anthropic's strategy has been to exaggerate the harmful capabilities of LLMs and systems driven by LLMs, positioning Anthropic themselves as the "safest" option. Take from this what you will. Yeah, I've been feeling that as well. It's not a bad strategy at all, makes sense, good for business. But on the nuclear issue, it's not a good sign that he's explicitly saying that this AGI future is a threat to nuc…
Earlier quoted context omitted.
What makes you think that they will just keep improving? It's not obvious at all, we might soon hit a ceiling, if we haven not already - time will tell. There are lots of technologies that have been 99% done for decades; it might be the same here.
Even if the technology doesn't get better, just imagine a world where all our processes are documented in a way that a computer can repeat them. And modifying the process requires nothing more than plain English or language. What used to require specialized integration can now be accomplished by a generalized agent.