Live data from Hacker News

Claude’s C Compiler vs. GCC

harshanu.space

361–370 of 377 posts

Re: Claude’s C Compiler vs. GCC

#361

Earlier quoted context omitted.

>Here's my question: why did the files that you submitted name Mark Shinwell as the author? >Beats me. AI decided to do so and I didn't question it. Haha that's comedy gold, and honestly a good interview screening situation - you'd instantly pass on the candidate!

I once had a PR. I told the dev that "LLM is ok but you own the code" He told me "I spent n days to architect the solution" He shows me claude generated system design .. and then i say ok, I went to review the code. 1hr later i asked why did you repeat the code all over at the end. Dude replies "junk the entire PR it's AI generated"

Has anyone who's familiar with compiler source code tried to compare it to other compilers? Given that LLMs have been trained on data sets that include the source code for numerous C compilers, is this just (say) pcc extruded in Rust form?

Re: Claude’s C Compiler vs. GCC

#362
post #178

One missing analysis, that IMHO is the most important right now , is : what is the quality of the generated code ? Having LLM generates a first complete iteration of a C compiler in rust is super useful if the code is of good enough quality that it can be maintained and improved by humans (or other AIs). It is (almost) completely useless otherwise. And that is the case for most of today's code generated by AIs. Most…

> with a clear structure, good naming, and sensible comments.

Additionally there is the additional problem, that LLM comments often represent what the code would be supposed to do, not what it actually does. People write comments to point out what was weird during implementation and what they found out during testing the implementation. LLM comments seems more to reflect the information present before writing the implementation, i.e. the use it as an internal check list what to generate.

In my opinion deceiving comments are worse than no comments at all.

Re: Claude’s C Compiler vs. GCC

#363

Can someone explain to me, what’s the big deal about this? The AI model was trained on lots of code and spit out sonething similar than gcc. Why is this revolutionary?

If someone told you 5 years ago that a computer generated a working C compiler, would you think it was a big deal or not?

These tools do not compete against the lonely programmer that writes everything from scratch they compete with the existing tooling. 5 years ago compiler generators already exist, as they did in the previous decades. That is a solved problem. People still like the handroll their parsers, not because generating wouldn't work, but because it has other benefits (maintainability, adaption, better diagnostics). Perfectly fine working code is routinely thrown away and reimplemented, because there are not enough people around anymore who know the code by heart. "The big Rewrite" is a meme for a reason.

Re: Claude’s C Compiler vs. GCC

#364
post #357

Earlier quoted context omitted.

>without a human babysitting it. From the original blog post: >Every agent would hit the same bug, fix that bug, and then overwrite each other's changes. Having 16 agents running didn't help because each was stuck solving the same task. >The fix was to use GCC as an online known-good compiler oracle to compare against. I wrote a new test harness that randomly compiled most of the kernel using GCC The blog post used t…

> The blog post used the word autonomous a lot, which I suppose is true if Nicholas Carlini is not a human being but in fact a Claude agent. Nothing in the article suggests it did not autonomously do the work. > Why would anyone do that? Because a lot of naysayers here pretend as if this is somehow trivial. > My point was that why does the company _not_ make a useful tool? Useful to whom? This is a researcher testing…

> Nothing in the article suggests it did not autonomously do the work.

I don’t know how to respond to that other than to ask you to quote the part of the blog post where the author described the language model running into a problem that it could not fix and then described the details of how he manually intervened to fix the problem that the language model could not fix when you elaborate on your definition of “nothing” in that sentence.

>Every agent would hit the same bug, fix that bug, and then overwrite each other's changes. Having 16 agents running didn't help because each was stuck solving the same task.

>The fix was to use GCC as an online known-good compiler oracle to compare against. I wrote a new test harness that randomly compiled most of the kernel using GCC

As for:

> Because a lot of naysayers here pretend as if this is somehow trivial.

This is an answer to “why do you want someone to do that?” You have already established that you would like that to happen. It doesn’t answer “why would a real human being (who is not you) that isn’t impressed by the compiler that doesn’t work put their time into making Anthropic look good?”

For example “I will pay a naysayer $20,000 to try” or “I know a guy that will pay a naysayer to try this, succeed or fail” or “I will give a naysayer a bunch of hardware to play with in exchange for attempting this” would be motivation to work for Anthropic and not get paid by Anthropic. Saying “I want you to do that because I think you’ll feel bad and waste your time” and then getting no takers isn’t really an assault on “the naysayers” decision not to do work for Anthropic without getting paid by Anthropic.

As for this, that’s a good question but I would say the bare minimum would be “useful”

> What, exactly would such a tool that'd somehow make the people dismissing this change their minds look like?

It is pretty common for tech companies to release free useful software. For example pytorch, react, Hack/hhvm etc. from Meta

https://opensource.fb.com/projects/

Or chromium from Google. Chromium is a good example, there’s a decent chance that you’re using a chromium based browser to read this. There’s also a ton of other stuff, golang comes to mind as another example.

https://opensource.google/

Or if you want stuff made by a business that’s a fraction the valuation of Anthropic, there’s Campfire and Writebook by 37signals. https://once.com/

> Because that wasn't the purpose.

I know that. That was the premise of my question.

I saw that they put a bunch of resources into making something that is not useful and asked why they did not put a bunch of resources into that was useful. Surely they could make something that is both useful and made their model look good?

For me it seems like the obvious answer would be either that they can’t make something useful:

> Their goal was to test the limits of what the model can achieve. They did that.

Or they don’t want to

> Because that wasn't the purpose.

I was asking if anyone had any substantive knowledge or informed opinion about whether it was one or the other but it seems like you’re saying it’s… both? They don’t want to make and release a useful tool and also they can not make and release a useful tool because this compiler, which is not useful, is the limit of what their model can achieve.

Like you want us all to know that they cannot and do not want to make any sort of useful tool. That is your clearly-stated opinion about their desires and capabilities. And also you want these “naysayers”, who are not you, to put their time and effort into… also not making something useful? To prove… what?

Re: Claude’s C Compiler vs. GCC

#365
It is a very controversial topic imo. I get that claude devs wants to show that their llm is capable of such a tedious task as building a compiler... But Pro-LLM people don't really get the idea of LLM.

Disclaimer: I have a near-zero competence in compilers and compiler-building but i just want to summarize what's going on in my opinion.

It's the same thing if i was given millions of repos of already-built compiler and had an ability to only wield these parts together. Yeah, it TECHNICALLY will work, but what's the point of building on top of the garbage afterwards?

You'll definitely want to refactor it, and it will not really be a pleasant experience to begin with. You have to have a certain amount of dedication and knowledge to contribute to this compiler, which you don't have if you're a plain vibe-coder. The things that are most difficult part of c compilers (and basically any compiler whatsoever) are optimizations and portability. Will you be able to have these things in a full claude-generated repo? Who knows! Maybe you'll cause an irreversible damage to the system of the end user, no one knows! There are so many snippets of code in the world, and you can't just filter-out the malicious and stupid ones.

The thing is, LLM's are stupid. I partially agree with Richard Stallman's take on current AI state - these are not intelligence, more of a bullshit generators if improperly used. Well to think, humans are partially LLM's themselves, but we have much more than that. LLM can only be used as a tool to help developers. My bet - never in the future the LLMs will be able to supply 100% prod-ready code by themselves. They are just not capable of that, it's in their nature to mimic and not to think.

LLMs in education and fast information fetching are blessing. It's the best thing that's happened since the invention of search engines. But never in my life will i blindly copypaste some shell-script or code that i don't know is not harmful or the code snippet lacks hyperlink to the original snippet of the code.

Vibe-coders imo are guys that copy-pasted stuff from internet back in... well, anytime since 2000s. They just evolved into guys that blindly copy-paste average result of their requests given by more convenient search engines. Not that it's a bad evolution step, it's just pretty much the same thing, but maybe it's less harmful to copypasters themselves.

THE BAD THING in CCC's creation is that some non-technical people are degenerates that will take this repo and say "LOOK, A COMPILER BUILT BY AN AI. AI!!! IT'S LIKE... A REALLY TEDIOUS TASK TO BUILD A COMPILER YKNOW. AND IT WAS BUILT (wielded from others people's repos) BY AI WITH NO HUMAN INTERVENTION. AND IT WORKS!!!!". No, it kind of doesn't. It even lacks "--help" lol. With every update, every pull request there is no guarantee that it will not become such an unstable codebase that any of its future extensions will either fail or misbehave. AI is only an option when ruled by the one who knows their stuff. They'll look at the code and say - well, that part is crappy, we need to refactor it", or "hey, that snippet is pretty good, didn't know you can do it that simple".

LLMs are just a big dictionary that you can either use to expand your knowledge about certain things you're interested into or to just blindly look for stuff you urgently need to use it once. If you want to ask somebody polish if you can borrow their phone, you certainly can grab Polish language dictionary, go to the part with sentences and read aloud: "Czy może skorzystac z twojego telefonu?". Will it help you learn? Technically yes, realistically - absolutely not. These snippets are only useful if you know how to use them right, how to form something with meaning out of them.

Pro-LLM people are dumb. But so are the Anti-LLM peoples. And what i mean by that is not "WE NEED AI EVERYWHERE!", but to acknowledge AI as a tool, not the worker.

As post-scriptum i want to add one thing - Pro-LLM mindset is a lot worse than Anti-LLM. AI guys, don't you see that the Bubble has already grown and becomes bigger and bigger as we go on? AI integration as of today is a really dum and frightening process. When you want to debate with Pro-LLM folks, please, don't act all high and mighty, you're not really in the situation to forbid someone from using something, especially CEOs, ESPECIALLY CEOs. With this attitude you're only contributing to building a wall with echo chamber for vibe coders. Monkey (ceo) see AI is capable of building something - monkey fire an entire department to save money on development team. Is the end result worse? Yes. But does it really bother mister Monkey - no, for him it's his another win for company's profit. He will not hear your point of view if you won't prove him the opposite - and yet again, you cannot do this if you're gonna act like you he doesn't know shit in business. It's literally the same thing that's happened to tons of job positions prior in human history, but with one small change - now it's tech, and every businessman thinks they knows tech because they use technical devices (idk, his smarthone or pc). BUSINESS DEMANDS PROFIT RAISE - always has been. You're gonna stand for your right to only integrate with AI wisely, not pushing it everywhere, and it is really important that you know how to do it.

If you're capable of boosting yourself with a bit of AI - why not? Performance boost will bend the learning curve in your favor, you only gonna win from that. And when the bubble will pop, the demand for real workers who know their stuff and who know how to boost themselves with right tools will skyrocket. That is, my bet.

Re: Claude’s C Compiler vs. GCC

#366

Earlier quoted context omitted.

Your comment is a perfect example of the biases I'm talking about. "AI evangelists" are not a singular group of people.

You're right! Sometimes even the right-here-right-now claims of AI capabilities are horseshit too with people in actuality remotely controlling the product. It's not common for present-capabilities to be lied about too. But it does happen! And the smallest presence are the users who don't work in the AI industry but rave about AI. I know of...two....people who fit that bill. A lead developer at a cybersecurity firm a…

Now you're just presenting straw man arguments and continuing to conflate separate issues.

Re: Claude’s C Compiler vs. GCC

#367

Earlier quoted context omitted.

You're right! Sometimes even the right-here-right-now claims of AI capabilities are horseshit too with people in actuality remotely controlling the product. It's not common for present-capabilities to be lied about too. But it does happen! And the smallest presence are the users who don't work in the AI industry but rave about AI. I know of...two....people who fit that bill. A lead developer at a cybersecurity firm a…

Now you're just presenting straw man arguments and continuing to conflate separate issues.

No no - I am agreeing with you that AI evangelists are not a singular group of people! Im fact I agreed so much I even gave a couple examples of how they're not a monolithic group!

Re: Claude’s C Compiler vs. GCC

#368
post #305

Earlier quoted context omitted.

"autonomously" I couldn't agree with, I use it regularly for 100-200 loc size stuff, I can't recall it ever being right the first time.

I regularly has it produce 10k+ lines of code that is working and passing extensive test suites. If you give it a prompt and no agent loop and test harness, then sure, you'll need to waste your time babysitting it.

If you give it a test harness then you're doing TDD? That will only work if you know what you're building, which is seldom the case.

Re: Claude’s C Compiler vs. GCC

#369
post #357

Earlier quoted context omitted.

> The blog post used the word autonomous a lot, which I suppose is true if Nicholas Carlini is not a human being but in fact a Claude agent. Nothing in the article suggests it did not autonomously do the work. > Why would anyone do that? Because a lot of naysayers here pretend as if this is somehow trivial. > My point was that why does the company _not_ make a useful tool? Useful to whom? This is a researcher testing…

> Nothing in the article suggests it did not autonomously do the work. I don’t know how to respond to that other than to ask you to quote the part of the blog post where the author described the language model running into a problem that it could not fix and then described the details of how he manually intervened to fix the problem that the language model could not fix when you elaborate on your definition of “nothi…

> I don’t know how to respond to that other than to ask you to quote the part of the blog post where the author described the language model running into a problem that it could not fix and then described the details of how he manually intervened to fix the problem that the language model could not fix when you elaborate on your definition of “nothing” in that sentence.

I suggest you re-read that and pay attention to how they're describing addressing these things by fixing the harness rather than solving the problems.

You conveniently quoted the part that doesn't support your claim.

> This is an answer to “why do you want someone to do that?” You have already established that you would like that to happen. It doesn’t answer “why would a real human being (who is not you) that isn’t impressed by the compiler that doesn’t work put their time into making Anthropic look good?”

My bad for assuming the naysayer care about learning something or understanding the technology rather than looking for excuses to ignorantly bash it.

> It is pretty common for tech companies to release free useful software. For example pytorch, react, Hack/hhvm etc. from Meta

You entirely failed to address my question.

> I saw that they put a bunch of resources into making something that is not useful and asked why they did not put a bunch of resources into that was useful. Surely they could make something that is both useful and made their model look good?

They made something that made their model look good to the people who are actually likely to want to use their model. Aka, the customers actually providing the vast majority of their revenue.

> I was asking if anyone had any substantive knowledge or informed opinion about whether it was one or the other but it seems like you’re saying it’s… both? They don’t want to make and release a useful tool and also they can not make and release a useful tool because this compiler, which is not useful, is the limit of what their model can achieve.

No, I've said there's no value in it for them to spend money on tools that'd just get dismissed and that at the same time wouldn't provide useful data to their actual customers.

> Like you want us all to know that they cannot and do not want to make any sort of useful tool.

I've said nothing of the sort. Stop lying about what I've said.

> That is your clearly-stated opinion about their desires and capabilities.

Another lie.

> And also you want these “naysayers”, who are not you, to put their time and effort into… also not making something useful? To prove… what?

I'd like them to stop being blatantly intellectually dishonest and make trite unjustified claims because they don't understand why this project had value, and actually try to learn and inform themselves.

It's naive to assume there's any honest curiosity lurking behind these shallow dismissals, I know, but I try to think the best of people until they prove their intent.

Re: Claude’s C Compiler vs. GCC

#370
post #305

Earlier quoted context omitted.

I regularly has it produce 10k+ lines of code that is working and passing extensive test suites. If you give it a prompt and no agent loop and test harness, then sure, you'll need to waste your time babysitting it.

If you give it a test harness then you're doing TDD? That will only work if you know what you're building, which is seldom the case.

TDD does not require you to know everything you're building up-front. Tests can come out of experimentation, to validate the final build. Tests can be driven by autonomous directed planning.

I'm currently, in fact, working on a system where the LLM semi-independently build up an understanding of a project and its goals from exploration, and then creates small targeted improvement plans, including the acceptance criteria that then feeds into building test suites which the build will finally be measured against.

It still needs direction - if you have a large spec or a judge/fitness function, such as you would for a compiler for an existing language, you can achieve a lot just from using that and may not need much additional direction. But even for far more exploratory projects, you can have the LLM surface perceived goals and plans to meet those goals, and "teach it" on the way by giving it points on how to revise a given goal or plan, and have e.g. implementation successes and failures feed into future plans.

My current system has "learned" [1] quite quickly on fairly complex test projects, and I'm in fact right now testing it on a hobby compiler project. The first cycles are frustrating (and an area I'm refining), because it's dumped into a project it doesn't know the real motivations for, and it will start making some code changes you know are bad, and letting go obsessing over that is hard. But ultimately using it as input to a feedback cycle where you add to its goals (e.g. make clear one of the goals is code that meets your specific standards) is more useful than managing it in detail yourself.

I'm very closet to putting this improvement agent in a cron job for a project I rely on for day to day use (yes, I'll make sure I can roll back), because it now very consistently implements improvements both entirely unilaterally, or based on minor hints (it has access to some files on my desktop, including a "journal" of sorts, and if I put a one-liner about an idea or frustration, I'll often come back to find a 300+ line implementation plan for a change to fix it, or lay the foundation for fixing it.

[1] "Learned" in this instance is in quotes for a reason. I'm not fine-tuning models - I have the agent do a retro of its own plan executions, and update documents with "lessons learned" that gets fed into the next planning stage.

Post reply on HN