Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We tasked Opus 4.6 using agent teams to build a C Compiler

anthropic.com

Post reply on HN
Print view View original

641–650 of 787 posts

Re: We tasked Opus 4.6 using agent teams to build a C Compiler

#641

Post by ajross »

vuciuc wrote: post #636
ajross wrote: post #617

Earlier quoted context omitted.

> Not for general purpose use, only for demo. What does that mean, though? I mean, it's already meeting a very high quality bar by booting at all and passing those tests. No, it doesn't beat existing solutions on all the checkboxes, but that's not what the demo is about. The point being demonstrated is that if you need a "custom compiler" or something similar for your own new, greenfield requirement , you can have it…

> you can have it at pretty-clearly-near-shippable quality in two weeks for $20k. if you spend months writing a tight spec, tests and have a better version of the compiler around to use when everything else fails.

> if you spend months writing a tight spec, tests and have a better version of the compiler around to use when everything else fails.

Doesn't matter because your competitors will have beaten you to market. That's just a simple Darwinian point, no AI magic needed.

No one doubts that things will be different in the coming Claudepocalypse, and new ideas about quality and process will need to happen to manage it. But sticking our heads in the sand and pretending that our stone tools are still better is just a path to early retirement at this point.

Re: We tasked Opus 4.6 using agent teams to build a C Compiler

#642

Post by boring-human »

smithcoin wrote: post #628
boring-human wrote: post #453

People focused on the flaws are missing the picture. Opus wasn't even trained to be "a member of a team of engineers," it was adapted to the task by one person with a shell script loop. Specific training for this mode of operation is inevitable. And model "IQ" is increasing with every generation. If human IQ is increasing at all, it's only because the engineer pool is shrinking more at one end than the other. This is…

> This is a five-alarm fire if you're a SWE and not retiring in the next couple years. I’m sorry, but this is such a hype beast take. In my opinion this is equivalent to telling people not to learn to drive five years ago because of self driving from Tesla. How is that going? Every single line of code produced is a liability. This idea that you’re going to have “gas town” like agents running and building apps without…

> In my opinion this is equivalent to telling people not to learn to drive five years ago because of self driving

Self-driving has different economics. We're reading tea leaves, true, but it's also true that software has zero marginal cost and that $20K pays for an engineer-month in SF.

> Every single line of code produced is a liability.

Do you have a hard spec and rock-solid test cases? If you do, you have two options to a working prototype: 2-6 engineer-years, or $20K. The second option will greatly increase in quality and likely decrease in price over the next few years.

What if the spec and the test cases are the new software? Assembly programmers used to make an argument against compiled code that's somewhat parallel to yours: every instruction is a (performance) liability.

> without humans in the loop

There will be humans, just fewer and fewer. The spec and test cases are AI-eligible too.

> safe, secure, and compliant software

I'm not sure humans' advantage here is safe, if it even exists still.

Re: We tasked Opus 4.6 using agent teams to build a C Compiler

#643

Post by marklsnyder »

Very cool, but I can't help but wonder how this translates to similarly complex projects where innate knowledge about the domain hasn't been embedded in the LLM via training data. There's a wealth of open source compiler code and related research papers that have been fed to the LLM. It seems like that would advantage the LLM significantly.

Re: We tasked Opus 4.6 using agent teams to build a C Compiler

#644

Post by cardanome »

qarl wrote: post #332
ndesaulniers wrote: post #165

I spent a good part of my career (nearly a decade) at Google working on getting Clang to build the linux kernel. https://clangbuiltlinux.github.io/ This LLM did it in (checks notes): > Over nearly 2,000 Claude Code sessions and $20,000 in API costs It may build, but does it boot (was also a significant and distinct next milestone)? (Also, will it blend?). Looks like yes! > The 100,000-line compiler can build a bootab…

> Still a really cool project! Yeah. This test sorta definitely proves that AI is legit. Despite the millions of people still insisting it's a hoax. The fact that the optimizations aren't as good as the 40 year gcc project? Eh - I think people who focus on that are probably still in some serious denial.

If will write you an C compiler by hand for 19k and it will be better than what Claude made.

Writing a toy C compiler isn't that hard. Any decent programmer can write one in a few weeks or months. The optimizations are the actually interesting part and Claude fails hard at that.

Re: We tasked Opus 4.6 using agent teams to build a C Compiler

#645

Post by disgruntledphd2 »

thesz wrote: post #605
disgruntledphd2 wrote: post #577

Earlier quoted context omitted.

> Do we know how many attempts were done to create such compiler before during previous tests? Would Anthropic report on the failed attempt? Can this "really, really impressive" thing be a result of a luck? No we don't and yeah we would expect them to only report positive results (this is both marketing and investigation). That being said, they provide all the code et al for people to review. I do agree that an out o…

> No we don't and yeah we would expect them to only report positive results (this is both marketing and investigation). This is matter of methodology. If they train models on that task or somewhat score/select models on their progress on that task, then we have test set leakage [1]. [1] https://en.wikipedia.org/wiki/Leakage_(machine_learning) This question is extremely important because test set leakage leads to impr…

> This is matter of methodology. If they train models on that task or somewhat score/select models on their progress on that task, then we have test set leakage [1].

I am quite familiar with leakage, having been building statistical models for maybe 15+ years at this point.

However, that's not really relevant in this particular case given that LLMs are trained on approximately the entire internet, so leakage is not really a concern (as there is no test set, apart from the tasks they get asked to do in post-training).

I think that's its impressive that this even works at all as even if it's just predicting tokens (which is basically what they're trained to), as this is a pointer towards potentially more useful tasks (convert this cobol code base to java, for instance).

I think the missing bit here is that this only works for cases where there's a really large test set (the html spec, the linux kernel). I'm not convinced that the models would be able to maintain coherence without this, so maybe that's what we need to figure out how to build to make this actually works.

Re: We tasked Opus 4.6 using agent teams to build a C Compiler

#646

Post by phendrenad2 »

marklsnyder wrote: post #643

Very cool, but I can't help but wonder how this translates to similarly complex projects where innate knowledge about the domain hasn't been embedded in the LLM via training data. There's a wealth of open source compiler code and related research papers that have been fed to the LLM. It seems like that would advantage the LLM significantly.

Not just open-source compilers, but books on compiler design, which have proliferated because every CS professor wants to take a crack at the problem.

Re: We tasked Opus 4.6 using agent teams to build a C Compiler

#647

Post by 8-prime »

adgjlsfhk1 wrote: post #451
travisgriggs wrote: post #450

A C Compiler seems like one of the more straightforward things to have done. Reading this gives me the same vibe as when a magician does a frequently done trick (saw someone in half, etc). I'd be more interested in letting it have a go at some some of the other "less trodden" paths of computing. Some of the things that would "wow me more": - Build a BEAM alternative, perhaps in an embedded space - Build a Smalltalk V…

if it actually compiles real C correctly, it's pretty impressive. The C standard is a total mess.

Yet we have gcc and clang navigating that mess. From which Opus 4.6 was able to take inspiration.

Re: We tasked Opus 4.6 using agent teams to build a C Compiler

#648

Post by sc68cal »

unglaublich wrote: post #597
iberator wrote: post #470

Earlier quoted context omitted.

Claude did not wrote it. you wrote it with PREVIOUS EXPERIENCE with 20.000 long commandshyellihg him exactly what to do. Real usable AI would create it with simple: 'make c compilers c99 faster than GCC'. AI usage should be banned in general. It takes jobs faster than creating new ones ..

Jobs are a means, not a goal.

Jobs are the only way that you survive in this society (food, shelter). Look how we treat unhoused people without jobs. AI is taking jobs away and that is putting people's survival at risk.

Re: We tasked Opus 4.6 using agent teams to build a C Compiler

#649

Post by thesz »

dcre wrote: post #608
thesz wrote: post #479

Earlier quoted context omitted.

> the huge gains in coding performance in the past year have come from RL, not from new sources of training data. This one was on HN recently: https://spectrum.ieee.org/ai-coding-degrades Author attributes past year's degradation of code generation by LLMs to excessive use of new source of training data, namely, users' code generation conversations.

Yeah, this is a bullshit article. There is no such degradation, and it’s absurd to say so on the basis of a single problem which the author describes as technically impossible. It is a very contrived under-specified prompt. And their “explanation” blaming the training data is just a guess on their part, one that I suspect is wrong. There is no argument given that that’s the actual cause of the observed phenomenon. It…

  > Yeah, this is a bullshit article. There is no such degradation, and it’s absurd to say so on the basis of a single problem which the author describes as technically impossible. It is a very contrived under-specified prompt.
I see "No True Scotsman" argument above.

  > My evidence is that RL is more relevant is that that’s what every single researcher and frontier lab employee I’ve heard speak about LLMs in the past year has said.
Reinforcement learning reinforces what is already in the LM, makes width of search path of possible correct answer narrower and wider search path in not-RL-tuned base models results in more correct answers [1].

[1] https://openreview.net/forum?id=4OsgYD7em5

  > I have never once heard any of them mention new sources of pretraining data, except maybe synthetic data they generate and verify themselves, which contradicts the author’s story because it’s not shitty code grabbed off the internet.
The sources of training data already were the reasons for allegations, even leading to lawsuits. So I would suspect that no engineer from any LLM company would disclose anything on their sources of training data besides innocently sounding "synthetic data verified by ourselves."

From the days I have worked on blockchains, I am very skeptical about any company riding any hype. They face enormous competition and they will buy, borrow or steal their way to try to not go down even a little. So, until Anthropic opens the way they train their model so that we can reproduce their results, I will suspect they leaked test set into it and used users code generation conversation as new source of training data.

Re: We tasked Opus 4.6 using agent teams to build a C Compiler

#650

Post by qarl »

lelanthran wrote: post #594
qarl wrote: post #589

Earlier quoted context omitted.

[flagged]

> Too late friend, you've revealed your stripes. So you are not willing to put $20k in escrow for, as per your offer: >>>> Deal. I'll pay you IF you can achieve the same level of performance. Heck, I'll double it. I just noticed now that you actually offered double. I will do it. This is my real name, my contact details are not hard to find. I will do it, with emitted binaries performing as well as or better than the…

HA.
Post reply on HN