What I find to be the most impressive part here is that it wrote the compiler without reference to the C specification and without architecture manuals at hand.
We tasked Opus 4.6 using agent teams to build a C Compiler
531–540 of 787 posts
Re: We tasked Opus 4.6 using agent teams to build a C Compiler
#532Can it create employment? How is this making life better. I understand the achievement but come on, wouldn´t it be something to show if you created employment for 10000 people using your 20000 USD! Microsoft, OpenAI, Anthropic, XAI, all solving the wrong problems, your problems not the collective ones.
Didn't you hear? We're heading towards a workless utopia where everything will be free (according to people who are actively working to eliminate things like food assistance for less fortunate mothers and children.)
Who are some of those people?
Re: We tasked Opus 4.6 using agent teams to build a C Compiler
#533Earlier quoted context omitted.
"It's like if a squirrel started playing chess and instead of "holy shit this squirrel can play chess!" most people responded with "But his elo rating sucks""
"The squirrel can do my job and more? It can do five years of my work in a month? For only $20k? Pssh, but I bet it copied someone's homework." Developer salaries are about to tank. This is the end of the line. People are just in denial. Soon companies will hire the squirrel instead of you. And the squirrel will transform into enormous infrastructure we can't afford ourselves. "One mega squirrel to implement your own…
This is really questionable outcome. So you'll have your own custom OS riddled with holes that AI won't be capable of fixing because the context and complexity became so high that running any small bug fix would cost thousands of dollars in tokens.
Is this how tech field ends? Overengineered brittle black-box monstrosities that nobody understands because important thing for business was "it does A, B, and C" and it doesn't matter how.
Re: We tasked Opus 4.6 using agent teams to build a C Compiler
#534Earlier quoted context omitted.
Yep. Building a working C compiler that compiles Linux is an impossible task for all but the top 1% of developers. And the ones that could do it have better things to do, plus they’d want a lot more than 20K for the trouble.
Interesting, why impossible? We studied compiler construction at uni. I might have to dig out a few books, but I’m confident I could write one. I can’t imagine anyone on my course of 120 nerds being unable to do this.
You are underestimating the complexity of the task so do other people on the thread. It's not trivial to implement a working C compiler very much so to implement the one that proves its worth by successfully compiling one of the largest open-source code repositories ever, which btw is not even a plain ISO C dialect.
Re: We tasked Opus 4.6 using agent teams to build a C Compiler
#535Earlier quoted context omitted.
It's amazing that it "works", but viability is another issue. It cost $20,000 and it worked, but it's also totally possible to spend $20,000 and have Claude shit out a pile of nonsense. You won't know until you've finished spending the money whether it will fail or not. Anthropic doesn't sell a contract that says "We'll only bill you if it works" like you can get from a bunch of humans. Do catastrophic bugs exist in…
> It cost $20,000 I'm curious - do you have ANY idea what it costs to have humans write 100,000 lines of code??? You should look it up. :)
That's irrelevant in this context, because it's not "get the humans to make a working product OR get the AI to make a working product"
The problem is you may pay $20K for gibberish, then try a second time, fail again, and then hire humans.
Coincidentally yes, I am aware, my last contract was building out a SCADA module the AI failed to develop at the company that contracted me.
I'm using that money to finance a new software company, and so far, AI hasn't been much help getting us off the ground.
Edit: oh yeah, and on top of paying Claude to fuck it up, you still have to also pay the salary of the guy arguing with Claude.
Re: We tasked Opus 4.6 using agent teams to build a C Compiler
#536Earlier quoted context omitted.
Nobody disputes that the LLM was drawing on knowledge in its training data. Obviously it was! But you'll need to be a bit more specific with your critique, because there is a whole spectrum of interpretations, from "it just decompressed fuzzily-stored code verbatim from the internet" (obviously wrong, since the Rust-based C compiler it wrote doesn't exist on the internet) all the way to "it used general knowledge fro…
The result is a fuzzy reproduction of the training input, specifically of the compilers contained within. The reproduction in a different, yet still similar enough programming language does not refute that. The implementation was strongly guided by a compiler and a suite of tests as an explicit filter on those outputs and limiting the acceptable solution space, which excluded unwanted interpolations of the training s…
> The result is a fuzzy reproduction of the training input, specifically of the compilers contained within.
Is it? I'm somewhat familiar with gcc and clang's source and it doesn't really particularly look like it to me.
https://github.com/anthropics/claudes-c-compiler/blob/main/s...
https://llvm.org/doxygen/LoopStrengthReduce_8cpp_source.html
https://github.com/gcc-mirror/gcc/blob/master/gcc/gimple-ssa...
Re: We tasked Opus 4.6 using agent teams to build a C Compiler
#537Most of the effort when writing a compiler is handling incorrect code, and reporting sensible error messages. Compiling known good code is a great start though.
Re: We tasked Opus 4.6 using agent teams to build a C Compiler
#538My first reaction: wow, incredible. My second reaction: still incredible, but noting that a C compiler is one of the most rigorously specified pieces of software out there. The spec is precise, the expected behavior is well-defined, and test cases are unambiguous. I'm curious how well this translates to the kind of work most of us do day-to-day where requirements are fuzzy, many edge cases are discovered on the go, a…
The C spec is certainly not formal or precise.
https://www.ralfj.de/blog/2020/12/14/provenance.html
Another example is that it's unclear from the standard if you can write malloc() in C.
Re: We tasked Opus 4.6 using agent teams to build a C Compiler
#539Earlier quoted context omitted.
> $20,000 set on fire To be fair, that's two weeks of the employer cost of a FAANG engineer's labor. And no human hacks a working compiler in two weeks. It's a lot of AI compute for a demo, sure. But $20k stunts are hardly unique. Clearly there's value being demonstrated here.
Yes a human can hack together a compiler in two weeks. If you can't, you should turn off the AI and learn for yourself for a while. Writing a compiler is not a flex; it's a couple very well understood problems, most of which can be solved using existing libraries. Parsing is solved with yacc, bison, or sitting down and writing a recursive descent parser (works for most well designed languages you can think of). Then…
Meanwhile:
> I spent a good part of my career (nearly a decade) at Google working on getting Clang to build the linux kernel.
Re: We tasked Opus 4.6 using agent teams to build a C Compiler
#540Earlier quoted context omitted.
> Still a really cool project! Yeah. This test sorta definitely proves that AI is legit. Despite the millions of people still insisting it's a hoax. The fact that the optimizations aren't as good as the 40 year gcc project? Eh - I think people who focus on that are probably still in some serious denial.
> This test sorta definitely proves that AI is legit. This is an "in distribution" test. There are a lot of C compilers out there, including ones with git history, implemented from scratch. "In distribution" tests do not test generalization. The "out of distribution" test would be like "implement (self-bootstrapping, Linux kernel compatible) C compiler in J." J is different enough from C and I know of no such compile…
> This is an "in distribution" test. There are a lot of C compilers out there, including ones with git history, implemented from scratch. "In distribution" tests do not test generalization.
It's still really, really impressive though.
Like, economics aside this is amazing progress. I remember GPT3 not being able to hold context for more than a paragraph, we've come a long way since then.
Hell, I remember bag of words being state of the art when I started my career. We have come a really, really, really long way since then.