Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We tasked Opus 4.6 using agent teams to build a C Compiler

anthropic.com

Post reply on HN
Print view View original

181–190 of 787 posts

Re: We tasked Opus 4.6 using agent teams to build a C Compiler

#181

Post by hn_acc1 »

NitpickLawyer wrote: post #53
gmueckl wrote: post #44

Earlier quoted context omitted.

The result is hardly a clean room implementation. It was rather a brute force attempt to decompress fuzzily stored knowledge contained within the network and it required close steering (using a big suite of tests) to get a reasonable approximation to the desired output. The compression and storage happened during the LLM training. Prove this statement wrong.

> Prove this statement wrong. If all it takes is "trained on the Internet" and "decompress stored knowledge", then surely gpt3, 3.5, 4, 4.1, 4o, o1, o3, o4, 5, 5.1, 5.x should have been able to do it, right? Claude 2, 3, 4, 4.1, 4.5? Surely.

Are you really asking for "all the previous versions were implemented so poorly they couldn't even do this simple, basic LLM task"?

Re: We tasked Opus 4.6 using agent teams to build a C Compiler

#182

Post by bwfan123 »

dmitrygr wrote: post #4

> The generated code is not very efficient. Even with all optimizations enabled, it outputs less efficient code than GCC with all optimizations disabled. Worse than "-O0" takes skill... So then, it produced something much worse than tcc (which is better than gcc -O0), an equivalent of which one man can produce in under two weeks. So even all those tokens and dollars did not equal one man's week of work. Except the on…

> I can already feel the contracts coming to fix LLM slop

First, the agents will attempt to fix issues on their own. Most easy problems will be fixed or worked-around in this manner. The hard problems will require a deeper causal model of how things work. For these, the agents will give up. But, the code-base has evolved to a point where no-one understands whats going on including the agents and its human handlers. Expect your phone to ring at that point, and prepare to ask for a ransom.

Re: We tasked Opus 4.6 using agent teams to build a C Compiler

#184

Post by password4321 »

akrauss wrote: post #69

I would like to see the following published: - All prompts used - The structure of the agent team (which agents / which roles) - Any other material that went into the process This would be a good source for learning, even though I'm not ready to spend 20k$ just for replicating the experiment.

Yes unfortunately these days most are satisfied with just the sausage and no details about how it was made.

Re: We tasked Opus 4.6 using agent teams to build a C Compiler

#185

Post by FeepingCreature »

jesse__ wrote: post #119
Marha01 wrote: post #94

Earlier quoted context omitted.

Even with 1 TB of weights (probable size of the largest state of the art models), the network is far too small to contain any significant part of the internet as compressed data, unless you really stretch the definition of data compression.

This sounds very wrong to me. Take the C4 training dataset for example. The uncompressed, uncleaned, size of the dataset is ~6TB, and contains an exhaustive English language scrape of the public internet from 2019. The cleaned (still uncompressed) dataset is significantly less than 1TB. I could go on, but, I think it's already pretty obvious that 1TB is more than enough storage to represent a significant portion of t…

This would imply that the English internet is not much bigger than 20x the English Wikipedia.

That seems implausible.

Re: We tasked Opus 4.6 using agent teams to build a C Compiler

#186

Post by FeepingCreature »

kgeist wrote: post #174
Marha01 wrote: post #94

Earlier quoted context omitted.

Even with 1 TB of weights (probable size of the largest state of the art models), the network is far too small to contain any significant part of the internet as compressed data, unless you really stretch the definition of data compression.

A lot of the internet is duplicate data, low quality content, SEO spam etc. I wouldn't be surprised if 1 TB is a significant portion of the high-quality, information-dense part of the internet.

I would be extremely surprised if it was that small.

Re: We tasked Opus 4.6 using agent teams to build a C Compiler

#188

Post by hn_acc1 »

0xCMP wrote: post #151
gmueckl wrote: post #44

Earlier quoted context omitted.

The result is hardly a clean room implementation. It was rather a brute force attempt to decompress fuzzily stored knowledge contained within the network and it required close steering (using a big suite of tests) to get a reasonable approximation to the desired output. The compression and storage happened during the LLM training. Prove this statement wrong.

I challenge anyone to try building a C compiler without a big suite of tests. Zig is the most recent attempt and they had an extensive test suite. I don't see how that is disqualifying. If you're testing a model I think it's reasonable that "clean room" have an exception for the model itself. They kept it offline and gave it a sandbox to avoid letting it find the answers for itself. Yes the compression and storage ha…

The point is - for a NEW project, no one has an extensive test suite. And if an extensive test suite exists, it's probably because the product that uses it also exists, already.

If it could translate the C++ standard INTO an extensive test suite that actually captures most corner cases, and doesn't generate false positives - again, without internet access and without using gcc as an oracle, etc?

Re: We tasked Opus 4.6 using agent teams to build a C Compiler

#189

Post by NitpickLawyer »

stephc_int13 wrote: post #167
geooff_ wrote: post #158

Maybe I'm naive, but I find these re-engineering complex product posts underwhelming. C Compilers exist and realistically Claudes training corpus contains a ton of C Compiler code. The task is already perfectly defined. There exists a benchmark of well-adopted codebases that can be used to prove if this is a working solution. Half the difficulty in making something is proving it works and is complete. IMO a simpler n…

This is a nice benchmark IMO. I would be curious to see how competitors and improved models would compare.

And how long will it take before an open model recreates this. The "vibe" consensus before "thinking" models really took off was that open was ~6mo behind SotA. With the massive RL improvements, over the past 6 months I've thought the gap was actually increasing. This will be a nice little verifiable test going forward.

Re: We tasked Opus 4.6 using agent teams to build a C Compiler

#190

Post by phillmv »

ndesaulniers wrote: post #165

I spent a good part of my career (nearly a decade) at Google working on getting Clang to build the linux kernel. https://clangbuiltlinux.github.io/ This LLM did it in (checks notes): > Over nearly 2,000 Claude Code sessions and $20,000 in API costs It may build, but does it boot (was also a significant and distinct next milestone)? (Also, will it blend?). Looks like yes! > The 100,000-line compiler can build a bootab…

i mean… your work also went into the training set, so it's not entirely surprising that it spat a version back out!
Post reply on HN