Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We tasked Opus 4.6 using agent teams to build a C Compiler

anthropic.com

Post reply on HN
Print view View original

301–310 of 787 posts

Re: We tasked Opus 4.6 using agent teams to build a C Compiler

#301

Post by uywykjdskn »

softwaredoug wrote: post #267
itay-maman wrote: post #168

My first reaction: wow, incredible. My second reaction: still incredible, but noting that a C compiler is one of the most rigorously specified pieces of software out there. The spec is precise, the expected behavior is well-defined, and test cases are unambiguous. I'm curious how well this translates to the kind of work most of us do day-to-day where requirements are fuzzy, many edge cases are discovered on the go, a…

Yes I think any codegen with a lot of tests and verification is more about “fitting” to the tests. Like fitting an ML model. It’s model training, not coding. But a lot of programming we discover correctness as we go, one reason humans don’t completely exit the loop. We need to see and build tests as we go, giving them particular care and attention to ensure they test what matters.

The agent can obviously do that

Re: We tasked Opus 4.6 using agent teams to build a C Compiler

#302

Post by signatoremo »

nicoburns wrote: post #272
libraryofbabel wrote: post #123

Earlier quoted context omitted.

Nobody disputes that the LLM was drawing on knowledge in its training data. Obviously it was! But you'll need to be a bit more specific with your critique, because there is a whole spectrum of interpretations, from "it just decompressed fuzzily-stored code verbatim from the internet" (obviously wrong, since the Rust-based C compiler it wrote doesn't exist on the internet) all the way to "it used general knowledge fro…

"clean room" usually means "without looking at the source code" of other similar projects. But presumably the AIs training data would have included GCC, Clang, and probably a dozen other C compilers.

Suppose you the human are working on a clean room implementation of C compiler, how do you go about doing it? Will you need to know about: a) the C language, and b) the inner working of a compiler? How did you acquire that knowledge?

Re: We tasked Opus 4.6 using agent teams to build a C Compiler

#303

Post by davemp »

Brute forcing a problem with a perfect test oracle and a really good heuristic (how many c compilers are in the training data) is not enough to justify the hype imo.

Yes this is cool. I actually have worked on a similar project with a slightly worse test oracle and would gladly never have to do that sort of work again. Just tedious unfulfilling work. Though we caught issues with both the specifications/test oracle when doing the work. Also many of the team members learned and are now SMEs for related systems.

Is this evidence that knowledge work is dead or AGI is coming? Absolutely not. I think you’d be pretty ignorant with respect to the field to suggest such a thing.

Re: We tasked Opus 4.6 using agent teams to build a C Compiler

#304

Post by signatoremo »

shakna wrote: post #212
NitpickLawyer wrote: post #53

Earlier quoted context omitted.

> Prove this statement wrong. If all it takes is "trained on the Internet" and "decompress stored knowledge", then surely gpt3, 3.5, 4, 4.1, 4o, o1, o3, o4, 5, 5.1, 5.x should have been able to do it, right? Claude 2, 3, 4, 4.1, 4.5? Surely.

Well, "Reimplement the c4 compiler - C in four functions" is absolutely something older models can do. Because most are trained, on that quite small product - its 20kb. But reimplementing that isn't impressive, because its not a clean room implementation if you trained on that data, to make the model that regurgitates the effort.

> Well, "Reimplement the c4 compiler - C in four functions" is absolutely something older models can do.

Are you sure about that? Do you have some examples? The older Claude models can’t do it according to TFA.

Re: We tasked Opus 4.6 using agent teams to build a C Compiler

#305

Post by andai »

nomel wrote: post #298
jbjbjbjb wrote: post #286

Earlier quoted context omitted.

It’s cool but there’s a good chance it’s just copying someone else’s homework albeit in an elaborate round about way.

I would claim that LLMs desperately need proprietary code in their training, before we see any big gains in quality. There's some incredible source available code out there. Statistically, I think there's a LOT more not so great source available code out there, because the majority of output of seasoned/high skill developers is proprietary. To me, a surprising portion of Claude 4.5 output definitely looks like studen…

Let's start with the source code for the Flash IDE :)

Re: We tasked Opus 4.6 using agent teams to build a C Compiler

#306

Post by andai »

beambot wrote: post #221
ndesaulniers wrote: post #165

I spent a good part of my career (nearly a decade) at Google working on getting Clang to build the linux kernel. https://clangbuiltlinux.github.io/ This LLM did it in (checks notes): > Over nearly 2,000 Claude Code sessions and $20,000 in API costs It may build, but does it boot (was also a significant and distinct next milestone)? (Also, will it blend?). Looks like yes! > The 100,000-line compiler can build a bootab…

This is getting close to a Ken Thompson "Trusting Trust" era -- AI could soon embed itself into the compilers themselves.

The asymmetry will be between the frontier AI's ability to create exploits vs find them.

Re: We tasked Opus 4.6 using agent teams to build a C Compiler

#307

Post by eek2121 »

ndesaulniers wrote: post #165

I spent a good part of my career (nearly a decade) at Google working on getting Clang to build the linux kernel. https://clangbuiltlinux.github.io/ This LLM did it in (checks notes): > Over nearly 2,000 Claude Code sessions and $20,000 in API costs It may build, but does it boot (was also a significant and distinct next milestone)? (Also, will it blend?). Looks like yes! > The 100,000-line compiler can build a bootab…

Also: a large amount of folks seem to think Claude code is losing a ton of money. I have no idea where the final numbers land, however, if the $20,000 figure is accurate and based on some of the estimates I've seen, they could've hired 8 senior level developers at a quarter million a year for the same amount of money spent internally.

Granted, marketing sucks up far too much money for any startup, and again, we don't know the actual numbers in play, however, this is something to keep in mind. (The very same marketing that likely also wrote the blog post, FWIW).

Re: We tasked Opus 4.6 using agent teams to build a C Compiler

#308

Post by steveklabnik »

kelnos wrote: post #275
NitpickLawyer wrote: post #19

This is a much more reasonable take than the cursor-browser thing. A few things that make it pretty impressive: > This was a clean-room implementation (Claude did not have internet access at any point during its development); it depends only on the Rust standard library. The 100,000-line compiler can build Linux 6.9 on x86, ARM, and RISC-V. It can also compile QEMU, FFmpeg, SQlite, postgres, redis > I started by draf…

Honestly I don't find it that impressive. I mean, it's objectively impressive that it can be done at all, but it's not impressive from the standpoint of doing stuff that nearly all real-world users will want it to do. The C specification and Linux kernel source code are undoubtedly in its training data, as are texts about compilers from a theoretical/educational perspective. Meanwhile, I'm certain most people will ne…

> Or perhaps write a complier for a new language that someone just invented, after writing a first draft of a spec for it.

Hello, this is what I did over my Christmas break. I've been taking some time to do other things, but plan on returning to it. But this absolutely works. Claude has written far more programs in my language than I have.

https://rue-lang.dev/ if you want to check it out. Spec and code are both linked there.

Re: We tasked Opus 4.6 using agent teams to build a C Compiler

#310

Post by nmilo »

array_key_first wrote: post #269
raincole wrote: post #161

Earlier quoted context omitted.

It's not a clean-room implementation, but not because it's trained on the internet. It's not a clean-room implementation because of this: > The fix was to use GCC as an online known-good compiler oracle to compare against

If you read the entire GCC source code and then create a compatible compiler, it's not clean room. Which Opus basically did since, I'm assuming, its training set contained the entire source of GCC. So even if they were actively referencing GCC I think that counts.

What if you just read the entire GCC source code in school 15 years ago? Is that not clean room?
Post reply on HN