Earlier quoted context omitted.
It's a bit disappointing that people are still re-hashing the same "it's in the training data" old thing from 3 years ago. It's not like any LLM could 1for1 regurgitate millions of LoC from any training set... This is not how it works. A pertinent quote from the article (which is a really nice read, I'd recommend reading it fully at least once): > Previous Opus 4 models were barely capable of producing a functional c…
Because for all those projects, the effective solution is to just use the existing implementation and not launder code through an LLM. We would rather see a stab at fixing CVEs or implementing features in open source projects. Like the wifi situation in FreeBSD.
We tasked Opus 4.6 using agent teams to build a C Compiler
51–60 of 787 posts
Re: We tasked Opus 4.6 using agent teams to build a C Compiler
#52It's weird to see the expectation that the result should be perfect. All said and done, that its even possible is remarkable. Maybe these all go into training the next Opus or Sonnet and we start getting models that can create efficient compilers from scratch. That would be something!
A symptom of the increasing backlash against generative AI (both in creative industries and in coding) is that any flaw in the resulting product is predicate to call it AI slop, even if it's very explicitly upfront that it's an experimental demo/proof of concept and not the NEXT BIG THING being hyped by influencers. That nuance is dead even outside of social media.
AI companies set that expectation when their CEOs ran around telling anyone who would listen that their product is a generational paradigm shift that will completely restructure both labor markets and human cognition itself. There is no nuance in their own PR, so why should they benefit from any when their product can't meet those expectations?
Re: We tasked Opus 4.6 using agent teams to build a C Compiler
#53This is a much more reasonable take than the cursor-browser thing. A few things that make it pretty impressive: > This was a clean-room implementation (Claude did not have internet access at any point during its development); it depends only on the Rust standard library. The 100,000-line compiler can build Linux 6.9 on x86, ARM, and RISC-V. It can also compile QEMU, FFmpeg, SQlite, postgres, redis > I started by draf…
The result is hardly a clean room implementation. It was rather a brute force attempt to decompress fuzzily stored knowledge contained within the network and it required close steering (using a big suite of tests) to get a reasonable approximation to the desired output. The compression and storage happened during the LLM training. Prove this statement wrong.
> Prove this statement wrong.
If all it takes is "trained on the Internet" and "decompress stored knowledge", then surely gpt3, 3.5, 4, 4.1, 4o, o1, o3, o4, 5, 5.1, 5.x should have been able to do it, right? Claude 2, 3, 4, 4.1, 4.5? Surely.
Re: We tasked Opus 4.6 using agent teams to build a C Compiler
#54How much of this result is effectively plagiarized open source compiler code? I don't understand how this is compelling at all: obviously it can regurgitate things that are nearly identical in capability to already existing code it was explicitly trained on... It's very telling how all these examples are all "look, we made it recreate a shitter version of a thing that already exists in the training set".
Honestly, probably not a lot. Not that many C compilers are compatible with all of GCC's weird features, and the ones that are, I don't think are written in Rust. Hell, even clang couldn't compile the Linux kernel until ~10 years ago. This is a very impressive project.
Re: We tasked Opus 4.6 using agent teams to build a C Compiler
#55> This was a clean-room implementation (Claude did not have internet access at any point during its development); it depends only on the Rust standard library. The 100,000-line compiler can build Linux 6.9 on x86, ARM, and RISC-V. It can also compile QEMU, FFmpeg, SQlite, postgres, redis, and has a 99% pass rate on most compiler test suites including the GCC torture test suite. It also passes the developer's ultimate…
Interesting how the concept of a clean room implementation changes when the agent has been trained on the entire internet already
To the best of my knowledge, there's no Rust-based compiler that comes anywhere close to 99% on the GCC torture test suite, or able to compile Doom. So even if it saw the internals of GCC and a lot of other compilers, the ability to recreate this step-by-step in Rust is extremely impressive to me.
Re: We tasked Opus 4.6 using agent teams to build a C Compiler
#56> The generated code is not very efficient. Even with all optimizations enabled, it outputs less efficient code than GCC with all optimizations disabled. Worse than "-O0" takes skill... So then, it produced something much worse than tcc (which is better than gcc -O0), an equivalent of which one man can produce in under two weeks. So even all those tokens and dollars did not equal one man's week of work. Except the on…
Claude is only a few years old so we should compare it to a 3 year old human's C compiler
Claude contains the entire wisdom of the internet, such as it is.
Re: We tasked Opus 4.6 using agent teams to build a C Compiler
#57Earlier quoted context omitted.
I would love to see the commit log on this.
did this before i knew how to git, back in college. target was ARMv5
Great. Did your compiler support three different architectures (four, if you include x86 in addition to x86-64) and compile and pass the test suite for all of this software?
> Projects that compile and pass their test suites include PostgreSQL (all 237 regression tests), SQLite, QuickJS, zlib, Lua, libsodium, libpng, jq, libjpeg-turbo, mbedTLS, libuv, Redis, libffi, musl, TCC, and DOOM — all using the fully standalone assembler and linker with no external toolchain. Over 150 additional projects have also been built successfully, including FFmpeg (all 7331 FATE checkasm tests on x86-64 and AArch64), GNU coreutils, Busybox, CPython, QEMU, and LuaJIT.
Writing a C compiler is not that difficult, I agree. Writing a C compiler that can compile a significant amount of real software across multiple architectures? That's significantly more non-trivial.
Re: We tasked Opus 4.6 using agent teams to build a C Compiler
#58So it copied one of the C compilers? This was always possible but now you need to pay $1000 in API costs to Anthropic
Add a 0 and double it
|Over nearly 2,000 Claude Code sessions and $20,000 in API cost
Re: We tasked Opus 4.6 using agent teams to build a C Compiler
#59> The generated code is not very efficient. Even with all optimizations enabled, it outputs less efficient code than GCC with all optimizations disabled. Worse than "-O0" takes skill... So then, it produced something much worse than tcc (which is better than gcc -O0), an equivalent of which one man can produce in under two weeks. So even all those tokens and dollars did not equal one man's week of work. Except the on…
I'm trying to recall a quote. Some war where all defeats were censored in the news, possibly Paris was losing to someone. It was something along the lines of "I can't help but notice how our great victories keep getting closer to home". Last year I tried using an LLM to make a joke language, I couldn't even compile the compiler the source code was so bad. Before Christmas, same joke language, a previous version of Cl…
> but I can't help but notice how the battles keep getting closer to home
This has been true for all of (known) human history. I’m gonna go ahead and make another bold prediction: tech will keep getting better.
The issue with this blog post is it’s mostly marketing.
Re: We tasked Opus 4.6 using agent teams to build a C Compiler
#60Earlier quoted context omitted.
Interesting how the concept of a clean room implementation changes when the agent has been trained on the entire internet already
To the best of my knowledge, there's no Rust-based compiler that comes anywhere close to 99% on the GCC torture test suite, or able to compile Doom. So even if it saw the internals of GCC and a lot of other compilers, the ability to recreate this step-by-step in Rust is extremely impressive to me.
The impressiveness of converting C to Rust by any means is kind of contingent on how much unnecessary unsafe there is in the end result though.