Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We tasked Opus 4.6 using agent teams to build a C Compiler

anthropic.com

Post reply on HN
Print view View original

701–710 of 787 posts

Re: We tasked Opus 4.6 using agent teams to build a C Compiler

#701

Post by Rudybega »

thesz wrote: post #677
Rudybega wrote: post #666

Earlier quoted context omitted.

There are two compilers that can handle the Linux kernel. GCC and LLVM. Both are written in C, not Rust. It's "in distribution" only if you really stretch the meaning of the term. A generic C compiler isn't going to be anywhere near the level of rigour of this one.

There is tinycc, that makes it three compilers. There is a C compiler implemented in Rust from scratch: https://github.com/PhilippRados/wrecc/commits/master/?after=... (the very beginning of commit history) There are several C compilers written in Rust from scratch of comparable quality. We do not know whether Anthropic has a closed source C compiler written in Rust in their training data. We also do not know whether…

TinyCC can't compile a modern linux kernel. It doesn't support a ton of the extensions they use. That Rust compiler similarly can't do it.

Re: We tasked Opus 4.6 using agent teams to build a C Compiler

#702

Post by famouswaffles »

nozzlegear wrote: post #426
famouswaffles wrote: post #387

Earlier quoted context omitted.

Cool I guess. Kind of a meaningless statement yeah? Let's hit the bend, then we'll talk. Until then repeating, 'It's an S Curve guys and what's more, we're near the bend! trust me" ad infinitum is pointless. It's not some wise revelation lol.

> Kind of a meaningless statement yeah? If you say so. It's clear you think these marketing announcements are still "exponential improvements" for some reason, but hey, I'm not an AI hype beast so by all means keep exponentialing lol

I'm not asking you to change your belief. By all means, think we're just around the corner of a plateau, but like I said, your statement is nothing meaningful or profound. It's your guess that things are about to slow down, that's all. It's better to just say that rather than talking about S curves and bends like you have any more insight than OP.

Re: We tasked Opus 4.6 using agent teams to build a C Compiler

#703

Post by wavemode »

p-e-w wrote: post #445
lelanthran wrote: post #441

Earlier quoted context omitted.

> > It cost $20,000 > I'm curious - do you have ANY idea what it costs to have humans write 100,000 lines of code??? I'll bite - I can write you an unoptimised C compiler that emits assembly for $20k, and it won't be 100k lines of code (maybe 15k, the last time I did this?). It won't take me a week, though. I think this project is a good frame of reference and matches my experience - vibing with AI is sometimes more…

> I can write you an unoptimised C compiler that emits assembly for $20k You may be willing to sell your work at that price, but that’s not the market rate, to put it very mildly. Even 10 times that would be seriously lowballing in the realm of contract work, regardless of whether it’s “optimised” or not (most software isn’t).

No, you're overestimating how complex it is to write an unoptimized C compiler. C is (in the grand scheme of things) a very simple language to implement a compiler for.

The rate probably goes up if you ask for more and more standards (C11, C17, C23...) but it's still a lot easier than compilers for almost any other popular language.

Re: We tasked Opus 4.6 using agent teams to build a C Compiler

#704

Post by thesz »

disgruntledphd2 wrote: post #645
thesz wrote: post #605

Earlier quoted context omitted.

> No we don't and yeah we would expect them to only report positive results (this is both marketing and investigation). This is matter of methodology. If they train models on that task or somewhat score/select models on their progress on that task, then we have test set leakage [1]. [1] https://en.wikipedia.org/wiki/Leakage_(machine_learning) This question is extremely important because test set leakage leads to impr…

> This is matter of methodology. If they train models on that task or somewhat score/select models on their progress on that task, then we have test set leakage [1]. I am quite familiar with leakage, having been building statistical models for maybe 15+ years at this point. However, that's not really relevant in this particular case given that LLMs are trained on approximately the entire internet, so leakage is not r…

  > I think the missing bit here is that this only works for cases where there's a really large test set (the html spec, the linux kernel). I'm not convinced that the models would be able to maintain coherence without this, so maybe that's what we need to figure out how to build to make this actually works.
Take any language with compiler and several thousands of users and you have a plenty of tests that approximate spec inward and outward.

Here's, for example, VHDL tests suite for GHDL, open source VHDL compiler and simulator: https://github.com/ghdl/ghdl/tree/master/testsuite

The GHDL test suite is sufficient and general enough to develop a pretty capable clone, to my knowledge. To my knowledge, there is only one open source VHDL compiler and it is written in Ada. And, again, expertise to implement another one from scratch to train an LLM on it is very, very scarce - VHDL, being highly parallel variant of Ada, is quirky as hell.

So someone can test your hypothesis on the VHDL - agent-code a VHDL compiler and simulator in Rust so that it passes GHDL test suite. Would it take two weeks and $20,000 as with C? I don't know but I really doubt so.

Re: We tasked Opus 4.6 using agent teams to build a C Compiler

#705

Post by dcre »

thesz wrote: post #649
dcre wrote: post #608

Earlier quoted context omitted.

Yeah, this is a bullshit article. There is no such degradation, and it’s absurd to say so on the basis of a single problem which the author describes as technically impossible. It is a very contrived under-specified prompt. And their “explanation” blaming the training data is just a guess on their part, one that I suspect is wrong. There is no argument given that that’s the actual cause of the observed phenomenon. It…

> Yeah, this is a bullshit article. There is no such degradation, and it’s absurd to say so on the basis of a single problem which the author describes as technically impossible. It is a very contrived under-specified prompt. I see "No True Scotsman" argument above. > My evidence is that RL is more relevant is that that’s what every single researcher and frontier lab employee I’ve heard speak about LLMs in the past y…

That is not what No True Scotsman is. I’m pointing out a bad argument with weak evidence.

Re: We tasked Opus 4.6 using agent teams to build a C Compiler

#706

Post by qarl »

qarl wrote: post #332
ndesaulniers wrote: post #165

I spent a good part of my career (nearly a decade) at Google working on getting Clang to build the linux kernel. https://clangbuiltlinux.github.io/ This LLM did it in (checks notes): > Over nearly 2,000 Claude Code sessions and $20,000 in API costs It may build, but does it boot (was also a significant and distinct next milestone)? (Also, will it blend?). Looks like yes! > The 100,000-line compiler can build a bootab…

> Still a really cool project! Yeah. This test sorta definitely proves that AI is legit. Despite the millions of people still insisting it's a hoax. The fact that the optimizations aren't as good as the 40 year gcc project? Eh - I think people who focus on that are probably still in some serious denial.

[flagged]

Re: We tasked Opus 4.6 using agent teams to build a C Compiler

#707

Post by thesz »

dcre wrote: post #705
thesz wrote: post #649

Earlier quoted context omitted.

> Yeah, this is a bullshit article. There is no such degradation, and it’s absurd to say so on the basis of a single problem which the author describes as technically impossible. It is a very contrived under-specified prompt. I see "No True Scotsman" argument above. > My evidence is that RL is more relevant is that that’s what every single researcher and frontier lab employee I’ve heard speak about LLMs in the past y…

That is not what No True Scotsman is. I’m pointing out a bad argument with weak evidence.

  >>> It is a very contrived under-specified prompt.
No True Prompt can be such contrived and underspecified.

The article about degradation is a case study (single prompt), weakest of the studies in hierarchy of knowledge. Case studies are basis for further, more rigorous studies. And author took the time to test his assumptions and presented quite clear evidence that such degradation might be present and that we should investigate.

Re: We tasked Opus 4.6 using agent teams to build a C Compiler

#708

Post by qarl »

bee_rider wrote: post #623
qarl wrote: post #545

Earlier quoted context omitted.

Deal. I'll pay you IF you can achieve the same level of performance. Heck, I'll double it. You must provide the entire git history with small commits. I won't be holding my breath.

You seem to have doubled down on a bluff that was already called.

Naw. I got him to reveal himself, which was the whole point.

It's amazing what you can get people to do.

Re: We tasked Opus 4.6 using agent teams to build a C Compiler

#710

Post by signatoremo »

sarchertech wrote: post #396
signatoremo wrote: post #302

Earlier quoted context omitted.

Suppose you the human are working on a clean room implementation of C compiler, how do you go about doing it? Will you need to know about: a) the C language, and b) the inner working of a compiler? How did you acquire that knowledge?

Doesn’t matter how you gain general knowledge of compiler techniques as long as you don’t have specific knowledge of the implementation of the compiler you are reverse engineering. If you have ever read the source code of the compiler you are reverse engineering, you are by definition not doing a clean room implementation.

> you are by definition not doing a clean room implementation.

This makes no sense. Reverse engineering IS an application of clean room implementation. Citing Wikipedia:

“Clean-room design (also known as the Chinese wall technique) is the method of copying a design by reverse engineering and then recreating it without infringing any of the copyrights associated with the original design”

https://en.wikipedia.org/wiki/Clean-room_design

Post reply on HN