Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

When AI Builds Itself: Our progress toward recursive self-improvement

anthropic.com

Post reply on HN
Print view View original

231–240 of 738 posts

Re: When AI Builds Itself: Our progress toward recursive self-improvement

#231

Post by sunaurus »

lukan wrote: post #187
overgard wrote: post #173

So, regardless of whether or not Anthropic CAN create a self improving AI.. does anyone else feel like they shouldn't be allowed to? Or it at least needs to be strictly supervised..? Like, I don't actually think Anthropic can make the singularity any time soon, but I think even AI boosters have to admit doing this is creating a society-wide danger for the benefit of a very very small number of already-rich people.

"does anyone else feel like they shouldn't be allowed to?" No. Technical limitations aside, I doubt it could be contained, but will be leaked soon, so won't profit just a small number of ultra rich.

I dunno, I find it extremely unbelievable that we will get self-improving AGI which chooses to become a slave to humanity at all, ultra rich or otherwise.

Re: When AI Builds Itself: Our progress toward recursive self-improvement

#232

Post by overgard »

lukan wrote: post #187
overgard wrote: post #173

So, regardless of whether or not Anthropic CAN create a self improving AI.. does anyone else feel like they shouldn't be allowed to? Or it at least needs to be strictly supervised..? Like, I don't actually think Anthropic can make the singularity any time soon, but I think even AI boosters have to admit doing this is creating a society-wide danger for the benefit of a very very small number of already-rich people.

"does anyone else feel like they shouldn't be allowed to?" No. Technical limitations aside, I doubt it could be contained, but will be leaked soon, so won't profit just a small number of ultra rich.

Step 1: Wait for scary doomsday AI to be leaked, Step 2: ???, Step 3: Profit!!

Re: When AI Builds Itself: Our progress toward recursive self-improvement

#233

Post by keybored »

mweidner wrote: post #119

I fail to see how pursuing recursive self-improvement at full speed is compatible with Anthropic's stated goal of AI Safety. If nukes were not invented yet, would it really be a good idea to build and sell them as fast as possible (in peace time, no less)? I am not cynical enough to believe that Anthropic's warnings are pure marketing hype. Let's hope that it is instead overconfidence or the result of too much time t…

Such a massively valued company. And doubting them is cynicism? It’s rational(ism).

So either they lie or they are AI Zealots. Interesting times.

Re: When AI Builds Itself: Our progress toward recursive self-improvement

#234

Post by rush86999 »

aleqs wrote: post #40

Okay, so anthropic has amazing AI which supposedly writes most of their code and can continuously improve... meanwhile they have outages on a regular basis, and any kind of long-running work will now consistently hit 'API Error: Server is temporarily limiting requests'. Not sure of this is intentional to force a reduction of token usage, but at this point I need to build around these throttling limits and outages wit…

Just as you expected, I'm throwing in my harness. Please support: https://github.com/rush86999/atom

Re: When AI Builds Itself: Our progress toward recursive self-improvement

#235

Post by keeda »

aleqs wrote: post #74
jakobnissen wrote: post #55

Earlier quoted context omitted.

Their outages are probably not due to their code though. It’s probably their infrastructure that can’t keep up. So seeing failures of infrastructure doesn’t really tell you anything about how good or bad Anthropic makes use of their models.

That seems like an assumption based on basically nothing. There is a lot of code at the infra layer, and based on the stack choices for Claude code and based on how buggy and unreliable ~everything from anthropic is, it seems pretty bizarre to claim these issues are not related to their code.

There are other indications, however, like Anthropic paying through the nose for compute just months after Dario told Dwarkesh how hard it is to predict demand, or ChatGPT and Codex not quite having the same issues after Altman spent much-publicized years scrounging for trillion-dollars of capacity.

While I'm very bullish on Anthropic, I'm a bit wary about their IPO because it seems to me that they're filing now while their financials look good and before other trends like the decline of tokenmaxxing and their compute bills catch up.

Re: When AI Builds Itself: Our progress toward recursive self-improvement

#236

Post by squidsoup »

malfist wrote: post #104
torben-friis wrote: post #36

>A caveat: Lines of code is an imperfect measure, as it measures quantity over quality. So 8× lines of code/engineer/day in the second quarter of 2026 is almost certainly an overstatement of the true productivity gain. Nonetheless, it indicates an acceleration. At Anthropic, we don’t reward people for how many lines of code they write; rather, team members are producing more code simply because they’re using AI syste…

One of my co-workers just asked me to review his pull request that was all AI generated. 600 files were touched, over 40k lines of code added. I'm sure he thought that was a crowning achievement, proof that AI can enable 10X developers, after all, what engineer could write 40k lines of code in a week? I declined to review it, stating that I couldn't possibly vet 40k lines of code, and wouldn't put my reputation on th…

That's a process problem at your company - no developer should be proposing branches over 1k loc (or whatever your agreed tolerance threshold is) without a very good reason, vibe coded or not.

Re: When AI Builds Itself: Our progress toward recursive self-improvement

#238

Post by keybored »

mweidner wrote: post #119

I fail to see how pursuing recursive self-improvement at full speed is compatible with Anthropic's stated goal of AI Safety. If nukes were not invented yet, would it really be a good idea to build and sell them as fast as possible (in peace time, no less)? I am not cynical enough to believe that Anthropic's warnings are pure marketing hype. Let's hope that it is instead overconfidence or the result of too much time t…

Such a massively valued company. And doubting them is cynicism? It’s rational(ism).

So either they lie or they are AI Zealots. Interesting times.

Edit:

> > and the two people I knew who later joined Anthropic seem like the type to do it for the greater good instead of money.

There are three types of people. Pedestrians, investors, and “I know some of them, they wouldn’t lie”.

Re: When AI Builds Itself: Our progress toward recursive self-improvement

#239

Post by overgard »

asdfman123 wrote: post #186
overgard wrote: post #173

So, regardless of whether or not Anthropic CAN create a self improving AI.. does anyone else feel like they shouldn't be allowed to? Or it at least needs to be strictly supervised..? Like, I don't actually think Anthropic can make the singularity any time soon, but I think even AI boosters have to admit doing this is creating a society-wide danger for the benefit of a very very small number of already-rich people.

I think that's a valid point. You could very well be right. But we're discussing whether we should close the barn door while the horse is three miles down the road.

Only if you think LLMs are the horse (I don't think they are). If they're not, then we should be building a brick wall in front of that door and hiring a full time security guard to watch it.

I realize he's saying it for hype, but if the CEO of the company goes around talking about how scared he is of what they're creating, hey, lets just take Dario at his word and put in some strict regulation. He won't mind if they're really about safety. (they're not)

Besides, yes, the knowledge of how to build these systems is out there, but the cost of doing it is staggeringly high (ie you can't run a frontier AI lab in your garage). There's only a limited number of known entities that need to be managed, and you can stop "progress" in its tracks by cutting off the money firehose.

Re: When AI Builds Itself: Our progress toward recursive self-improvement

#240

Post by qwery »

airstrike wrote: post #112
f311a wrote: post #88

Earlier quoted context omitted.

Infrastructure is a much harder problem. They can't even improve Claude Code, which eats 1GB+ of RAM. Meanwhile, my editor only consumes 80MB of RAM.

This might explain it, in the opposite way it was meant to: https://fxtwitter.com/trq212/status/2014051501786931427 > Most people's mental model of Claude Code is that "it's just a TUI" but it should really be closer to "a small game engine".

~ "it's not a TUI! and my dad works at Nintendo"

curses, bud. curses.

It's genuinely difficult to tell how much of this is true. The post is obviously 100% posturing, but some of the words describe things that could be done.

Very few game engines do anything I'd describe as rasterisation. That's kind of the point of a GPU. Well, it used to be. I suppose "small game engines" might be more likely on average to include a rasteriser. The typical reason for this is because the author wanted to write it. Whereas big engine make triangle give hardware go brrr.

So I assume here 'rasterize' means 'printf'. And diffing screens means diffing 50..150 lines of text. And "generating ANSI sequences to draw" means 'printf' with some ANSI sequences interpolated in.

Then there's the frame budget. You have to understand they are operating within a strict frame budget -- they're not messing around, OK. They have a 16 ms frame budget, so they burned 11 ms and now have a (roughly) ~5 ms approx. budget for the final 'printf' in the chain???

Post reply on HN