Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

When AI Builds Itself: Our progress toward recursive self-improvement

anthropic.com

Post reply on HN
Print view View original

461–470 of 738 posts

Re: When AI Builds Itself: Our progress toward recursive self-improvement

#461

Post by rdw »

gensym wrote: post #175
mweidner wrote: post #119

I fail to see how pursuing recursive self-improvement at full speed is compatible with Anthropic's stated goal of AI Safety. If nukes were not invented yet, would it really be a good idea to build and sell them as fast as possible (in peace time, no less)? I am not cynical enough to believe that Anthropic's warnings are pure marketing hype. Let's hope that it is instead overconfidence or the result of too much time t…

> I am not cynical enough to believe that Anthropic's warnings are pure marketing hype. Nor am I. I think they believe that AI poses a grave danger, and they are playing the prisoner's dilemma as an unvirtuous actor. 1. If anyone builds strong AI, it may be catastrophically bad. 2. If anyone builds strong AI, it will be better for the builder than for anyone who does not. Either because it won't be catastrophically b…

I spoke with an Anthropic employee, and came to understand that their definition of safety is more like "making AI be a tool that humans can use without hurting themselves or others more than they can already do". It's literally about how AI makes it easier for people to construct bombs, poisons, manipulation, and exploits. Consistent with their caution about releasing Mythos to unvetted actors. So it's not about superintelligence killing humanity, at least as far as this employee conveyed to me.

This means their strategy is more like:

1. If someone builds a market-leading unsafe strong AI, it may be misused in a damaging way by a large number of humans, undermining society and creating a catastrophic upheaval.

2. However, if the leading AI maker also works to make it safe against misuse, as long as the stay in the lead and keep it safe, then the ability of human bad actors to misuse the AI is limited. Given enough time, society will adapt to pretty much anything, so eventually there's no longer an arms race to stay ahead.

I don't really know whether I agree with their concerns, but I do think that (my understanding of) their principles is that they're reasonable, self-consistent, and they adhere to them in all their public and private actions.

Re: When AI Builds Itself: Our progress toward recursive self-improvement

#462

Post by onlyrealcuzzo »

pizlonator wrote: post #416
onlyrealcuzzo wrote: post #348

Earlier quoted context omitted.

I'm building a memory safe programming language with a declarative concurrency model that's close to release. There is ZERO chance I would ever be able to complete it on my own. I doubt it'll get traction, but if it doesn't, I am pretty confident a future language will take the ideas for polymorphic synchronization and profile-guided optimization. It has an easy version/mode of compilation that makes Rust's affine ow…

How do you know it’s actually memory safe?

I have ~5500 memory safety fuzz tests, four different test suites with between ~80%-99% line/branch coverage each, and the same design as Rust, and haven't found a memory safety issue in 4 weeks, and I'm still planning another ~4 weeks of testing before release, more if need be.

Rust had memory safety bugs well after release - IIUC all the way until after the 1.0 release.

So, it's highly unlikely to be perfect, but I think it'll be in better shape than Go or Rust were when they initially launched.

Re: When AI Builds Itself: Our progress toward recursive self-improvement

#463

Post by asdff »

throwway120385 wrote: post #358
mapBasketWand wrote: post #333

Earlier quoted context omitted.

I love the idea of installing high resolution cameras in the fridge to monitor the ice maker to feed into a vision model that renders digital ice to the exact position of the real ice on the fridge’s giant screen

Or you could... open the door and look inside.

What, like a poor?

Re: When AI Builds Itself: Our progress toward recursive self-improvement

#464

Post by human305893 »

spprashant wrote: post #393
pizlonator wrote: post #282

What I can’t get over is that there have been exactly zero software breakthroughs since vibe coding started, other than vibe coding itself. Claude is amazing, that’s true. But if it was as amazing as this article implies, I’d expect some breakthrough outside of AI itself. Rewriting a Zig program in unsafe Rust? Not a breakthrough. Finding a bunch of security vulns? Maybe that’s sort of a breakthrough though it’s unde…

Its in a weird space right now. These models are actually extremely good but they are far from an intelligence unto themselves. Truth is if someone told you they could build these things 5 years ago, you d write them a check for a trillion dollars. Problem is once we got them, we realized they are not all that. Its like a mecha suit in a universe, where mecha suits are abundant and cheap. Someone has to climb into th…

Most of the skeptics exist because of the grandiose claims made by the AI companies saying pure hype marketing bs. If this was just a tool, discussed at the scope of what the tools can actually produce and do, there would be sensible discourse about it.

Re: When AI Builds Itself: Our progress toward recursive self-improvement

#465

Post by knollimar »

fluoridation wrote: post #258
jrflo wrote: post #97

Earlier quoted context omitted.

I think harnesses would count, AI != LLMs. Any piece of code that helps the computer reason for itself is AI, the harnesses are AI in a sense.

By that interpretation, neither the harness nor the LLM is the AI. The computer (or system of computers) taken as a whole is the AI. You can't remove any piece and still have an intelligent system.

Does this extend to power generation then, too?

Re: When AI Builds Itself: Our progress toward recursive self-improvement

#466

Post by ivraatiems »

Whether or not Anthropic is right about what AI can accomplish, whether these performance gains are real or not, their moral stance here is absolutely hideous to me.

"We must blast forwards into making this dangerous thing because if we don't, someone else surely will," is a coward's argument.

If you believe it is dangerous, you should be dedicating yourself to STOPPING others from making it, not making it first! There's a reason disarmament has been so important in nuclear politics! It's not because people think nukes are a great idea!

In fact, that kind of thinking is exactly what keeps nukes dangerous!

If they themselves buy what they're selling, they should shut the whole thing down. Fortunately, I don't think they do, and neither do I, yet.

Re: When AI Builds Itself: Our progress toward recursive self-improvement

#467

Post by AnimalMuppet »

hombre_fatal wrote: post #441
aleqs wrote: post #40

Okay, so anthropic has amazing AI which supposedly writes most of their code and can continuously improve... meanwhile they have outages on a regular basis, and any kind of long-running work will now consistently hit 'API Error: Server is temporarily limiting requests'. Not sure of this is intentional to force a reduction of token usage, but at this point I need to build around these throttling limits and outages wit…

This comment is a good example of the double standard laymen have about AI usage: If you use AI, then AI must be expected to solve all problems, even problems that affect everyone like infra scaling. And if perfection isn’t delivered, then of course it wasn’t: you used AI and AI sucks.

If their AI is good enough to write their code, why isn't it good enough to tell them how to fix their infra? That's a different problem space, but it's not harder than the code.

Re: When AI Builds Itself: Our progress toward recursive self-improvement

#468

Post by c-hendricks »

sothatsit wrote: post #374
pizlonator wrote: post #282

What I can’t get over is that there have been exactly zero software breakthroughs since vibe coding started, other than vibe coding itself. Claude is amazing, that’s true. But if it was as amazing as this article implies, I’d expect some breakthrough outside of AI itself. Rewriting a Zig program in unsafe Rust? Not a breakthrough. Finding a bunch of security vulns? Maybe that’s sort of a breakthrough though it’s unde…

Maybe my bar for what constitutes a breakthrough is lower than other people's, but all of these seem like breakthroughs to me: NLP as a field saw huge shifts. NLP tasks that used to be complex and inaccurate can now be setup very easily and quickly using structured outputs from LLMs, often with greater accuracy. A small charity I help with has now been able to build their own website to manage their day-to-day operat…

> ChatGPT has 1 billion MAU. People are now getting life advice, financial advice, and mental health help from chatbots

Personally not the kind of breakthrough I'm psyched about

Re: When AI Builds Itself: Our progress toward recursive self-improvement

#469

Post by jaggederest »

strix_varius wrote: post #420
AceJohnny2 wrote: post #274

Earlier quoted context omitted.

This is a decades-old design pattern when CPU >> IO. Emacs has been doing just that since the 80s, when people were complaining about "Eight Megs And Constantly Swapping". See "redisplay" [1] This minimizes screen flash. You can't rely on terminals doing double-buffering. [1] https://github.com/emacs-mirror/emacs/blob/c29071587c64efb30... or a more user-friendly overview, Daniel Colascione's seminal "Buttery Smooth E…

lol what? There are definitely ways to make non flashing terminal UIs without this total insanity.

ncurses (new curses) was "new" in 1993...

Re: When AI Builds Itself: Our progress toward recursive self-improvement

#470

Post by NewsaHackO »

square_usual wrote: post #402
emp17344 wrote: post #373

Earlier quoted context omitted.

Not if you’re claiming that the spells, once cast, automatically get exponentially spellier until they awaken into a spell god, capable of literally anything, including casting more complicated spells than any wizard is capable of. If that were true, you’d have no need for wizards. The fact that wizards are still around means it’s probably bullshit.

They literally aren't! they literally say in this article that it's not there yet!!!

Did actually expect people to read the article before commenting?
Post reply on HN