101–110 of 377 posts
What does the smallest (simplest in terms of complexity / lines of code) C-compiler that can compile and run SQLite look like? Perhaps that would be a more telling benchmark to evaluate the Claude compiler against.
I think this is a great example of both points of view in the ongoing debate. Pro-LLM coding agents: look! a working compiler built in a few hours by an agent! this is amazing! Anti-LLM coding agents: it's not a working compiler, though. And it doesn't matter how few hours it took, because it doesn't work. It's useless. Pro: Sure, but we can get the agent to fix that. Anti: Can you, though? We've seen that the more c…
> It's not fair to compare them like this! As someone who leans pro in this debate, I don't think I would make that statement. I would say the results are exactly as we expect. Also, a highly verifiable task like this is well suited to LLMs, and I expect within the next ~2 years AI tools will produce a better compiler than gcc.
That's what always puts me off: when AI replaces artists, SO and FOSS projects, it can only feed into itself and deteriorate..
The 158,000x slowdown on SQLite is the number that matters here, not whether it can parse C correctly. Parsing is the solved problem — every CS undergrad writes a recursive descent parser. The interesting (and hard) parts of a compiler are register allocation, instruction selection, and optimization passes, and those are exactly where this falls apart. That said, I think the framing of "CCC vs GCC" is wrong. GCC has…
This thing has likely all of GCC, clang and any other open source C compiler in its training set. It could have spotted out GCC source code verbatim and matched its performance.
If I had GCC and was asked for a C compiler I would just provide GCC..
Earlier quoted context omitted.
> Pro-LLM coding agents: look! a working compiler built in a few hours by an agent! this is amazing! > Anti-LLM coding agents: it's not a working compiler, though. And it doesn't matter how few hours it took, because it doesn't work. It's useless. Pro-LLM: Read the freaking article, it's not that long. The compiler made a mistake in an area where only two compilers exist that are up to the task: Linux Kernel.
Anti-LLM: isn’t all this intelligence supposed to give us something better than what we already have?
[1] There are around 8k programming languages that are used (or were used) in practice (that is, they were deemed better than existing ones in some aspects) and there are around 50 million programmers. I use it to estimate how many people did something, which is objectively better than existing products.
It seems like if Anthropic released a super cool and useful _free_ utility (like a compiler, for example) that was better than existing counterparts or solved a problem that hadn’t been solved before[0] and just casually said “Here is this awesome thing that you should use every day. By the way our language model made this.” it would be incredible advertising for them. But they instead made a blog post about how it w…
"Ironically, among the four stages, the compiler (translation to assembly) is the most approachable one for an AI to build. It is mostly about pattern matching and rule application: take C constructs and map them to assembly patterns. The assembler is harder than it looks. It needs to know the exact binary encoding of every instruction for the target architecture. x86-64 alone has thousands of instruction variants wi…
I am inclined to agree with you... but, did CC produce a working linker as well as a working compiler? I thought it was just the compiler that Anthropic produced.
I think this is a great example of both points of view in the ongoing debate. Pro-LLM coding agents: look! a working compiler built in a few hours by an agent! this is amazing! Anti-LLM coding agents: it's not a working compiler, though. And it doesn't matter how few hours it took, because it doesn't work. It's useless. Pro: Sure, but we can get the agent to fix that. Anti: Can you, though? We've seen that the more c…
As a neutral observation: it’s remarkable how quickly we as humans adjust expectations. Imagine five years ago saying that you could have a general purpose AI write a c compiler that can handle the Linux kernel, by itself, from scratch for $20k by writing a simple English prompt. That would have been completely unbelievable! Absurd! No one would take it seriously. And now look at where we are.
Just because we're here doesn't mean we're getting to AGI or software developers begging for jobs at Starbucks
I think this is a great example of both points of view in the ongoing debate. Pro-LLM coding agents: look! a working compiler built in a few hours by an agent! this is amazing! Anti-LLM coding agents: it's not a working compiler, though. And it doesn't matter how few hours it took, because it doesn't work. It's useless. Pro: Sure, but we can get the agent to fix that. Anti: Can you, though? We've seen that the more c…
"The source code of gcc is available online"
This is what I've noticed about most LLM generated code, its about the quality of an undergrad, and I think there's a good reason for this - most of the code its been trained on is of undergrad quality. Stack overflow questions, a lot of undergrad open source projects, there are some professional quality open source projects (eg SqlLite) but they are outweighed by the mass of other code. Also things like Sqllite don't compare to things like Oracle or Sql Server which are proprietary.