Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We tasked Opus 4.6 using agent teams to build a C Compiler

anthropic.com

Post reply on HN
Print view View original

481–490 of 787 posts

Re: We tasked Opus 4.6 using agent teams to build a C Compiler

#481

Post by pertymcpert »

sarchertech wrote: post #396
signatoremo wrote: post #302

Earlier quoted context omitted.

Suppose you the human are working on a clean room implementation of C compiler, how do you go about doing it? Will you need to know about: a) the C language, and b) the inner working of a compiler? How did you acquire that knowledge?

Doesn’t matter how you gain general knowledge of compiler techniques as long as you don’t have specific knowledge of the implementation of the compiler you are reverse engineering. If you have ever read the source code of the compiler you are reverse engineering, you are by definition not doing a clean room implementation.

Claude was not reverse engineering here. By your definition no one can do a clean room implementation if they've taken a recent compilers course at university.

Re: We tasked Opus 4.6 using agent teams to build a C Compiler

#482

Post by pjmlp »

qarl wrote: post #438
ThrowawayR2 wrote: post #429

Earlier quoted context omitted.

Anyone remember the dotcom bust?

Oh yeah, I do. That whole internet thing was a total HOAX. I can't believe people bought into that. Can you imagine if Amazon, EBay, PayPal, or Saleforce existed today?

Well, how is your Solaris installation going?

I also remember having gone into research, because there were no jobs available, and even though I was employed at the time, our salaries weren't being paid.

Re: We tasked Opus 4.6 using agent teams to build a C Compiler

#483

Post by D-Machine »

DiogenesKynikos wrote: post #477
D-Machine wrote: post #373

Earlier quoted context omitted.

To make some vague claims explicit here, for interested readers: > "We quantify the proportion of the ground-truth book that appears in a production LLM’s generated text using a block-based, greedy approximation of longest common substring (nv-recall, Equation 7). This metric only counts sufficiently long, contiguous spans of near-verbatim text, for which we can conservatively claim extraction of training data (Secti…

The one-shot performance of their recall attempts is much less impressive. The two best-performing models were only able to reproduce about 70% of a 1000-token string. That's still pretty good, but it's not as if they spit out the book verbatim. In other words, if you give an LLM a short segment of a very well known book, it can guess a short continuation (several sentences) reasonably accurately, but it will usually…

Right, and this should be contextualized with respect to code generation. It is not crazy to presume that LLMs have effectively nearly perfectly memorized certain training sources, but the ability to generate / extract outputs that are nearly identical to those training sources will of course necessarily be highly contingent on the prompting patterns and complexity.

So, dismissals of "it was just translating C compilers in the training set to Rust" need to be carefully quantified, but, also, need to be evaluated in the context of the prompts. As others in this post have noted, there are basically no details about the prompts.

Re: We tasked Opus 4.6 using agent teams to build a C Compiler

#484

Post by pertymcpert »

panzi wrote: post #196
NitpickLawyer wrote: post #19

This is a much more reasonable take than the cursor-browser thing. A few things that make it pretty impressive: > This was a clean-room implementation (Claude did not have internet access at any point during its development); it depends only on the Rust standard library. The 100,000-line compiler can build Linux 6.9 on x86, ARM, and RISC-V. It can also compile QEMU, FFmpeg, SQlite, postgres, redis > I started by draf…

> clean-room implementation Except its trained on all source out there, so I assume on GCC and clang. I wonder how similar the code is to either.

I'm familiar with both compilers. There's more similarity to LLVM, it even borrows some naming such as mem2reg (which doesn't really exist anymore) and GetElementPtr. But that's pretty much where things end. The rest of it is just common sense.

Re: We tasked Opus 4.6 using agent teams to build a C Compiler

#485

Post by pertymcpert »

LinXitoW wrote: post #413
dwaltrip wrote: post #149

Earlier quoted context omitted.

The CEOs aren't here in the comments.

Which is why we ought to always bring up their BS every time people try to pretend it didn't happen. The promises made are ABSOLUTELY relevant to how promising or not these experiments are.

I bet you get upset when you buy a new iPhone and don't love it, because Tim Cook said on the ad that they think you're going to love it.

Re: We tasked Opus 4.6 using agent teams to build a C Compiler

#486

Post by [deleted] »

tumdum_ wrote: post #346
PostOnce wrote: post #336

Earlier quoted context omitted.

It's amazing that it "works", but viability is another issue. It cost $20,000 and it worked, but it's also totally possible to spend $20,000 and have Claude shit out a pile of nonsense. You won't know until you've finished spending the money whether it will fail or not. Anthropic doesn't sell a contract that says "We'll only bill you if it works" like you can get from a bunch of humans. Do catastrophic bugs exist in…

> On top of that, Anthropic is losing money on it. It seems they are *not* losing money on inference: https://bsky.app/profile/steveklabnik.com/post/3mdirf7tj5s2e

[deleted]

Re: We tasked Opus 4.6 using agent teams to build a C Compiler

#487

Post by miohtama »

qarl wrote: post #332
ndesaulniers wrote: post #165

I spent a good part of my career (nearly a decade) at Google working on getting Clang to build the linux kernel. https://clangbuiltlinux.github.io/ This LLM did it in (checks notes): > Over nearly 2,000 Claude Code sessions and $20,000 in API costs It may build, but does it boot (was also a significant and distinct next milestone)? (Also, will it blend?). Looks like yes! > The 100,000-line compiler can build a bootab…

> Still a really cool project! Yeah. This test sorta definitely proves that AI is legit. Despite the millions of people still insisting it's a hoax. The fact that the optimizations aren't as good as the 40 year gcc project? Eh - I think people who focus on that are probably still in some serious denial.

GCC had 40 years headstart

Re: We tasked Opus 4.6 using agent teams to build a C Compiler

#489

Post by ndesaulniers »

ori_b wrote: post #207
ndesaulniers wrote: post #176

Earlier quoted context omitted.

> C compiler is one of the most rigorously specified pieces of software out there /me Laughs in "unspecified behavior."

There's undefined behavior, which is quite well specified. What do you mean by unspecified behavior? Do you have an example?

https://www.open-std.org/jtc1/sc22/wg14/www/docs/n3685.pdf

Read section J.1.

Re: We tasked Opus 4.6 using agent teams to build a C Compiler

#490

Post by 9rx »

ndesaulniers wrote: post #165

I spent a good part of my career (nearly a decade) at Google working on getting Clang to build the linux kernel. https://clangbuiltlinux.github.io/ This LLM did it in (checks notes): > Over nearly 2,000 Claude Code sessions and $20,000 in API costs It may build, but does it boot (was also a significant and distinct next milestone)? (Also, will it blend?). Looks like yes! > The 100,000-line compiler can build a bootab…

> I spent a good part of my career (nearly a decade) at Google working on getting Clang to build the linux kernel.

How much of that time was spent writing the tests that they found to use in this experiment? You (or someone like you) were a major contributor to this. All Opus had to do here was keep brute forcing a solution until the tests passed.

It is amazing that it is possible at all, but remains an impossibly without a heavy human hand. One could easily still spend a good part of their career reproducing this if they first had to rewrite all of the tests from scratch.

Post reply on HN