Reflections on AI at the End of 2025
antirez.com
Reflections on AI at the End of 2025
1–10 of 383 posts
Re: Reflections on AI at the End of 2025
#2> There are certain tasks, like improving a given program for speed, for instance, where in theory the model can continue to make progress with a very clear reward signal for a very long time.
This makes me think: I wonder if Goodhart's law[1] may apply here. I wonder if, for instance, optimizing for speed may produce code that is faster but harder to understand and extend. Should we care or would it be ok for AI to produce code that passes all tests and is faster? Would the AI become good at creating explanations for humans as a side effect?
And if Goodhard's law doesn't apply, why is it? Is it because we're only doing RLVR fine-tuning on the last layers of the network so all the generality of the pre-training is not lost? And if this is the case, could this be a limitation in not being able to be creative enough to come up with move 37?
Re: Reflections on AI at the End of 2025
#3Not sure I understand the last sentence:
> The fundamental challenge in AI for the next 20 years is avoiding extinction.
Re: Reflections on AI at the End of 2025
#4Not sure I understand the last sentence: > The fundamental challenge in AI for the next 20 years is avoiding extinction.
I think he's referring to AI safety.
https://lesswrong.com/posts/uMQ3cqWDPHhjtiesc/agi-ruin-a-lis...
Re: Reflections on AI at the End of 2025
#5Not sure I understand the last sentence: > The fundamental challenge in AI for the next 20 years is avoiding extinction.
He's referring to humanity, I believe
Re: Reflections on AI at the End of 2025
#6There's videos about Diffusion LLMs too, apparently getting rid of the linear token generation. But I'm no ML engineer.
Re: Reflections on AI at the End of 2025
#7> The fundamental challenge in AI for the next 20 years is avoiding extinction.
That's a weird thing to end on. Surely it's worth more than one sentence if you're serious about it? As it stands, it feels a bit like the fearmongering Big Tech CEOs use to drive up the AI stocks.
If AI is really that powerful and I should care about it, I'd rather hear about it without the scare tactics.
Re: Reflections on AI at the End of 2025
#8Not sure I understand the last sentence: > The fundamental challenge in AI for the next 20 years is avoiding extinction.
He's referring to humanity, I believe
It's ambiguous. It could go the other way. He could be referring to that oldest of science fiction tropes: The Bulterian Jihad, the human revolt against thinking machines.
Re: Reflections on AI at the End of 2025
#9> * The fundamental challenge in AI for the next 20 years is avoiding extinction.
This reminded me of the Don’t look up movie where they basically gambled with the humans extinction.
Re: Reflections on AI at the End of 2025
#10> The fundamental challenge in AI for the next 20 years is avoiding extinction. That's a weird thing to end on. Surely it's worth more than one sentence if you're serious about it? As it stands, it feels a bit like the fearmongering Big Tech CEOs use to drive up the AI stocks. If AI is really that powerful and I should care about it, I'd rather hear about it without the scare tactics.
I would say yes, everyone should care about it.
There is plenty of material on the topic. See for example https://ai-2027.com/ or https://www.lesswrong.com/posts/uMQ3cqWDPHhjtiesc/agi-ruin-a...