Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We tasked Opus 4.6 using agent teams to build a C Compiler

anthropic.com

Post reply on HN
Print view View original

491–500 of 787 posts

Re: We tasked Opus 4.6 using agent teams to build a C Compiler

#491

Post by viccis »

whinvik wrote: post #12

It's weird to see the expectation that the result should be perfect. All said and done, that its even possible is remarkable. Maybe these all go into training the next Opus or Sonnet and we start getting models that can create efficient compilers from scratch. That would be something!

>It's weird to see the expectation that the result should be perfect.

Given that they spent $20k on it and it's basically just advertising targeted at convincing greedy execs to fire as many of us as they can, yeah it should be fucking perfect.

Re: We tasked Opus 4.6 using agent teams to build a C Compiler

#492

Post by ip26 »

ndesaulniers wrote: post #165

I spent a good part of my career (nearly a decade) at Google working on getting Clang to build the linux kernel. https://clangbuiltlinux.github.io/ This LLM did it in (checks notes): > Over nearly 2,000 Claude Code sessions and $20,000 in API costs It may build, but does it boot (was also a significant and distinct next milestone)? (Also, will it blend?). Looks like yes! > The 100,000-line compiler can build a bootab…

I’m excited and waiting for the team that shows with $20k in credits they can substantially speed up the generated code by improving clang!

Re: We tasked Opus 4.6 using agent teams to build a C Compiler

#493

Post by krzat »

andrewshawcare wrote: post #397

It used the best tests it could find for existing compilers. This is effectively steering Claude to a well-defined solution. Hard to find fully specified problems like this in the wild. I think this is more a testament to small, well-written tests than it is agent teams. I imagine you could do the same thing with any frontier model and a single agent in a linear flow. I don’t know why people use parallel agents and i…

You know what else is well specified? LLM improving on itself.

Re: We tasked Opus 4.6 using agent teams to build a C Compiler

#495

Post by arcanemachiner »

iberator wrote: post #470
ndesaulniers wrote: post #165

I spent a good part of my career (nearly a decade) at Google working on getting Clang to build the linux kernel. https://clangbuiltlinux.github.io/ This LLM did it in (checks notes): > Over nearly 2,000 Claude Code sessions and $20,000 in API costs It may build, but does it boot (was also a significant and distinct next milestone)? (Also, will it blend?). Looks like yes! > The 100,000-line compiler can build a bootab…

Claude did not wrote it. you wrote it with PREVIOUS EXPERIENCE with 20.000 long commandshyellihg him exactly what to do. Real usable AI would create it with simple: 'make c compilers c99 faster than GCC'. AI usage should be banned in general. It takes jobs faster than creating new ones ..

That's actually pretty funny. They're patting it on the back for using, in all likelihood, some significant portions of code that they actually wrote, which was stolen from them without attribution so that it could be used as part of a very expensive parlour trick.

Re: We tasked Opus 4.6 using agent teams to build a C Compiler

#496

Post by grey-area »

ndesaulniers wrote: post #165

I spent a good part of my career (nearly a decade) at Google working on getting Clang to build the linux kernel. https://clangbuiltlinux.github.io/ This LLM did it in (checks notes): > Over nearly 2,000 Claude Code sessions and $20,000 in API costs It may build, but does it boot (was also a significant and distinct next milestone)? (Also, will it blend?). Looks like yes! > The 100,000-line compiler can build a bootab…

Isn't the AI basing what it does heavily on the publicly available source code for compilers in C though? Without that work it would not be able to generate this would it? Or in your opinion is it sufficiently different from the work people like you did to be classed as unique creation?

I'm curious on your take on the references the GAI might have used to create such a project and whether this matters.

Re: We tasked Opus 4.6 using agent teams to build a C Compiler

#497

Post by f311a »

Rudybega wrote: post #205
falloutx wrote: post #197

Earlier quoted context omitted.

Why does language of the compiler matter? Its a solved problem and since other implementations are already available anyone can already transpile them to rust.

Direct transpilation would create a ton of unsafe code (this repo doesn't have any) and fixing that would require a lot of manual fixes from the model. Even that would be a massive achievement, but it's not how this was created.

They are trained pretty hard to transpile the code between languages and do this pretty well because this can be done using RL.

You can force the agent not to use unsafe, this is why it burned $20000. Thousands of attempts against good tests with good boundaries set.

Re: We tasked Opus 4.6 using agent teams to build a C Compiler

#498

Post by f311a »

NitpickLawyer wrote: post #25
7734128 wrote: post #17

I'm sure this is impressive, but it's probably not the best test case given how many C compilers there are out there and how they presumably have been featured in the training data. This is almost like asking me to invent a path finding algorithm when I've been thought Dijkstra's and A*.

It's a bit disappointing that people are still re-hashing the same "it's in the training data" old thing from 3 years ago. It's not like any LLM could 1for1 regurgitate millions of LoC from any training set... This is not how it works. A pertinent quote from the article (which is a really nice read, I'd recommend reading it fully at least once): > Previous Opus 4 models were barely capable of producing a functional c…

That's because they still struggle hard with out-of-distribution tasks even though some of them can be solved using existing training data pretty well. Focusing on out-of-distribution will probably lower scores for benchmarks. They focus too much on common tasks.

And keep in mind, the original creators of the first compiler had to come up with everything: lexical analysis -> parsing -> IR -> codegen -> optimization. LLMs are not yet capable of producing a lot of novelty. There are many areas in compilers that can be optimized right now, but LLMs can't help with that.

Re: We tasked Opus 4.6 using agent teams to build a C Compiler

#499

Post by jillesvangurp »

btown wrote: post #26

> This was a clean-room implementation (Claude did not have internet access at any point during its development); it depends only on the Rust standard library. The 100,000-line compiler can build Linux 6.9 on x86, ARM, and RISC-V. It can also compile QEMU, FFmpeg, SQlite, postgres, redis, and has a 99% pass rate on most compiler test suites including the GCC torture test suite. It also passes the developer's ultimate…

You can use ai coding tools to create test suites, specifications, documentation, etc. And you can use them to scrutinize those, review them, criticize them, etc. Not having a test suite just means you start with creating one. Then the next question of course becomes "for what?".

This indeed puts human prompters in a position where their job is to set the goals, outline the vision, ask for the right things, ask critical questions, and to correct where needed.

Human contractors are a good analogy. Because they tend to come in without too much context into a new job. Their context is mainly what they've done before. But it takes time to get up to speed with whatever the customer is asking for and their context. People are slightly better at getting information out of other people. AI coding tools don't ask enough critical questions, yet. But that sounds fixable. The breakthroughs here are as much in the feedback loops and plumbing around the models as they are in the models themselves. It's all about getting the right information in and out of the context.

Post reply on HN