Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I read all of Cloudflare's Claude-generated commits

maxemitchell.com

Post reply on HN
Print view View original

71–80 of 291 posts

Re: I read all of Cloudflare's Claude-generated commits

#71

Post by a2128 »

Dylan16807 wrote: post #6
SupremumLimit wrote: post #4

It's an interesting review but I really dislike this type of techno-utopian determinism: "When models inevitably improve..." Says who? How is it inevitable? What if they've actually reached their limits by now?

Models are improving every day. People are figuring out thousands of different optimizations to training and to hardware efficiency. The idea that right now in early June 2025 is when improvement stops beggars belief. We might be approaching a limit, but that's going to be a sigmoid curve, not a sudden halt in advancement.

I think at this point we're reaching more incremental updates, which can score higher on some benchmarks but then simultaneously behave worse with real-world prompts, most especially if they were prompt engineered for a specific model. I recall Google updating their Flash model on their API with no way to revert to the old one and it caused a lot of people to complain that everything they've built is no longer working because the model is just behaving differently than when they wrote all the prompts.

Re: I read all of Cloudflare's Claude-generated commits

#72

Post by atomlib »

Sevii wrote: post #8
SupremumLimit wrote: post #4

It's an interesting review but I really dislike this type of techno-utopian determinism: "When models inevitably improve..." Says who? How is it inevitable? What if they've actually reached their limits by now?

Models have improved significantly over the last 3 months. Yet people have been saying 'What if they've actually reached their limits by now?' for pushing 3 years.

https://xkcd.com/605/

Re: I read all of Cloudflare's Claude-generated commits

#73

Post by never_inline »

> Don't be afraid to get your hands dirty. Some bugs and styling issues are faster to fix manually than to prompt through. Knowing when to intervene is part of the craft.

This has been my experience as well. to always run the cli tool in the bottom pane of an IDE and not in a standalone terminal.

Re: I read all of Cloudflare's Claude-generated commits

#74

Post by djtango »

mschild wrote: post #61
gizmo686 wrote: post #59

Earlier quoted context omitted.

My work has involved a project that is almost entirely generated code for over a decade. Not AI generated, the actual work of the project is in creating the code generator. One of the things we learned very quickly was that having generated source code in the same repository as actual source code was not sustainable. The nature of reviewing changes is just too different between them. Another thing we learned very qui…

> One of the things we learned very quickly was that having generated source code in the same repository as actual source code was not sustainable. Keeping a repository with the prompts, or other commands separate is fine, but not committing the generated code at all I find questionable at best.

I didn't read it as that - If I understood correctly, generated code must be quarantined very tightly. And inevitably you need to edit/override generated code and the manner by which you alter it must go through some kind of process so the alteration is auditable and can again be clearly distinguished from generated code.

Tbh this all sounds very familiar and like classic data management/admin systems for regular businesses. The only difference is that the data is code and the admins are the engineers themselves so the temptation to "just" change things in place is too great. But I suspect it doesn't scale and is hard to manage etc.

Re: I read all of Cloudflare's Claude-generated commits

#75

Post by never_inline »

SrslyJosh wrote: post #5

> Reading through these commits sparked an idea: what if we treated prompts as the actual source code? Imagine version control systems where you commit the prompts used to generate features rather than the resulting implementation. Please god, no, never do this. For one thing, why would you not commit the generated source code when storage is essentially free? That seems insane for multiple reasons. > When models ine…

Apart from obvious non-reproducibility, the other problem is lack of navigable structure. I can't command+click or "show usages" or "show definition" any more.

Re: I read all of Cloudflare's Claude-generated commits

#76

Post by chrishare »

Xelbair wrote: post #70
SrslyJosh wrote: post #5

> Reading through these commits sparked an idea: what if we treated prompts as the actual source code? Imagine version control systems where you commit the prompts used to generate features rather than the resulting implementation. Please god, no, never do this. For one thing, why would you not commit the generated source code when storage is essentially free? That seems insane for multiple reasons. > When models ine…

Worse. Models aren't deterministic! They use temperature value to control randomness, just so they can escape local minima! Regenerated code might behave differently, have different bugs(worst case), or not work at all(best case).

Nitpick - it's the ML system that is sampling from model predictions that has a temperature parameter, not the model itself. Temperature and even model aside, there are other sources of randomness like the underlying hardware that can cause the havoc you describe.

Re: I read all of Cloudflare's Claude-generated commits

#78

Post by auggierose »

fch42 wrote: post #67
dcre wrote: post #60

Earlier quoted context omitted.

Right. I don’t understand why everyone thinks this will make it impossible for junior devs to learn. The people I had around to answer my questions when I was learning knew a whole lot less than Claude and also had full time jobs doing something other than answering my questions.

It won't make it impossible for junior engineers to learn. It will simply reduce the amount of opportunities to learn (and not just for juniors), by virtue of companies' beancounters concluding "two for one" (several juniors) doesn't return the same as "buy one get one free" (existing staff + AI license). I dread the day we all "learn from AI". The social interaction part of learning is just as important as the conte…

I learnt programming on my own, without any social interaction involved. In fact, I loved programming because it does not involve any social interaction.

Programming has become more of a "social game" in the last 15 years or so. AI is a new superpower for people like me, bringing balance to the Force.

Re: I read all of Cloudflare's Claude-generated commits

#79

Post by someothherguyy »

tptacek wrote: post #58
kiitos wrote: post #57

Earlier quoted context omitted.

What on earth are you talking about?? If the LLM hallucinates, then the code it produces is wrong. That wrong code isn't obviously or programmatically determinable as wrong, the agent has no way to figure out that it's wrong, it's not as if the LLM produces at the same time tests that identify that hallucinated code as being wrong. The only way that this wrong code can be identified as wrong is by the human user "loo…

No, what's happening here is we're talking past each other. An agent lints and compiles code . The LLM is stochastic and unreliable. The agent is ~200 lines of Python code that checks the exit code of the compiler and relays it back to the LLM. You can easily fool an LLM. You can't fool the compiler. I didn't say anything about whether code needs to be reviewed line-by-line by humans. I review LLM code line-by-line.…

Linting isn't verification of correctness, and yes, you can fool the compiler, linters, etc. Work with some human interns, they are great at it. Agents will do crazy things to get around linting errors, including removing functionality.

Re: I read all of Cloudflare's Claude-generated commits

#80

Post by keybored »

tptacek wrote: post #44
deadbabe wrote: post #12

Earlier quoted context omitted.

5 years ago a person would be blown away by today’s LLMs. But people today will merely say “cool” at whatever LLMs are in use 5 years from now. Or maybe not even that.

Most of the developers I know personally who have been radicalized by coding agents, it happened within the past 9 months. It does not feel like we are in a phase of predictable boring improvement.

Radicalized? Going with the flow and wishes of the people who are driving AI is the opposite of that.

To have their minds changed drastically, sure..

Post reply on HN