Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I read all of Cloudflare's Claude-generated commits

maxemitchell.com

Post reply on HN
Print view View original

41–50 of 291 posts

Re: I read all of Cloudflare's Claude-generated commits

#41

Post by derwiki »

stingraycharles wrote: post #37
fastball wrote: post #30

Earlier quoted context omitted.

The idea as stated is a poor one, but a slight reshuffling and it seems promising: You generate code with LLMs. You write tests for this code, either using LLMs or on your own. You of course commit your actual code: it is required to actually run the program, after all. However you also save the entire prompt chain somewhere. Then (as stated in the article), when a much better model comes along, you re-run that chain…

Means the temperature should be set to 0 (which not every provider supports) so that the output becomes entirely deterministic. Right now with most models if you give the same input prompt twice it will give two different solutions.

Two years ago when I was working on this at a startup, setting OAI models’ temp to 0 still didn’t make them deterministic. Has that changed?

Re: I read all of Cloudflare's Claude-generated commits

#42

Post by fastball »

stingraycharles wrote: post #37
fastball wrote: post #30

Earlier quoted context omitted.

The idea as stated is a poor one, but a slight reshuffling and it seems promising: You generate code with LLMs. You write tests for this code, either using LLMs or on your own. You of course commit your actual code: it is required to actually run the program, after all. However you also save the entire prompt chain somewhere. Then (as stated in the article), when a much better model comes along, you re-run that chain…

Means the temperature should be set to 0 (which not every provider supports) so that the output becomes entirely deterministic. Right now with most models if you give the same input prompt twice it will give two different solutions.

I would only care about more deterministic output if I was repeating the same process with the same model, which is not the point of the exercise.

Re: I read all of Cloudflare's Claude-generated commits

#43

Post by tptacek »

declan_roberts wrote: post #22

These posts are funny to me because prompt engineers point at them as evidence of the fast-approaching software engineer obsolescence but the years of experience in software engineering necessary to even guide an AI in this way is very high. The reason he keeps adjusting the prompts is because he knows how to program. He knows what it should look like. It just blurs the line between engineer and tool.

I don't know why that's funny. This is not a post about a vibe coding session. It's Kenton Varda['s coding session].

later

updated to clarify kentonv didn't write this article

Re: I read all of Cloudflare's Claude-generated commits

#44

Post by tptacek »

deadbabe wrote: post #12
Dylan16807 wrote: post #6

Earlier quoted context omitted.

Models are improving every day. People are figuring out thousands of different optimizations to training and to hardware efficiency. The idea that right now in early June 2025 is when improvement stops beggars belief. We might be approaching a limit, but that's going to be a sigmoid curve, not a sudden halt in advancement.

5 years ago a person would be blown away by today’s LLMs. But people today will merely say “cool” at whatever LLMs are in use 5 years from now. Or maybe not even that.

Most of the developers I know personally who have been radicalized by coding agents, it happened within the past 9 months. It does not feel like we are in a phase of predictable boring improvement.

Re: I read all of Cloudflare's Claude-generated commits

#45

Post by tptacek »

greyadept wrote: post #25
Sevii wrote: post #8

Earlier quoted context omitted.

Models have improved significantly over the last 3 months. Yet people have been saying 'What if they've actually reached their limits by now?' for pushing 3 years.

For me, improvement means no hallucination, but that only seems to have gotten worse and I'm interested to find out whether it's actually solvable at all.

Why do you care about hallucination for coding problems? You're in an agent loop; the compiler is ground truth. If the LLM hallucinates, the agent just iterates. You don't even see it unless you make the mistake of looking closely.

Re: I read all of Cloudflare's Claude-generated commits

#46

Post by tptacek »

IncreasePosts wrote: post #18

I asked this in the other thread (no response, but I was a bit late) How does anyone using AI like this have confidence that they aren't unintentionally plagiarizing code and violating the terms of whatever license it was released under? For random personal projects I don't see it mattering that much. But if a large corp is releasing code like this, one would hope they've done some due diligence that they have to jus…

Most of the code generated by LLMs, and especially the code you actually keep from an agent, is mid, replacement-level, boring stuff. If you're not already building projects with LLMs, I think you need to start doing that first before you develop a strong take on this. From what I see in my own work, the code being generated is highly unlikely to be distinguishable. There is more of me and my prompts and decisions in the LLM code than there can possibly be defensible IPR from anybody else, unless the very notion of, like, wrapping a SQLite INSERT statement in Golang is defensible.

The best way I can explain the experience of working with an LLM agent right now is that it is like if every API in the world had a magic "examples" generator that always included whatever it was you were trying to do (so long as what you were trying to do was within the obvious remit of the library).

Re: I read all of Cloudflare's Claude-generated commits

#47

Post by kevingadd »

tptacek wrote: post #43
declan_roberts wrote: post #22

These posts are funny to me because prompt engineers point at them as evidence of the fast-approaching software engineer obsolescence but the years of experience in software engineering necessary to even guide an AI in this way is very high. The reason he keeps adjusting the prompts is because he knows how to program. He knows what it should look like. It just blurs the line between engineer and tool.

I don't know why that's funny. This is not a post about a vibe coding session. It's Kenton Varda['s coding session]. later updated to clarify kentonv didn't write this article

I think it makes sense that GP is skeptical of this article considering it contains things like:

> this tool is improving itself, learning from every interaction

which seem to indicate a fundamental misunderstanding of how modern LLMs work: the 'improving' happens by humans training/refining existing models offline to create new models, and the 'learning' is just filling the context window with more stuff, not enhancement of the actual model or the model 'learning' - it will forget everything if you drop the context and as the context grows it can 'forget' things it previously 'learned'.

Re: I read all of Cloudflare's Claude-generated commits

#48

Post by weird-eye-issue »

stingraycharles wrote: post #37
fastball wrote: post #30

Earlier quoted context omitted.

The idea as stated is a poor one, but a slight reshuffling and it seems promising: You generate code with LLMs. You write tests for this code, either using LLMs or on your own. You of course commit your actual code: it is required to actually run the program, after all. However you also save the entire prompt chain somewhere. Then (as stated in the article), when a much better model comes along, you re-run that chain…

Means the temperature should be set to 0 (which not every provider supports) so that the output becomes entirely deterministic. Right now with most models if you give the same input prompt twice it will give two different solutions.

Claude Code already uses a temperature of 0 (just inspect the requests) but it's not deterministic

Not to mention it also performs web searches, web fetching etc which would also make it not deterministic

Re: I read all of Cloudflare's Claude-generated commits

#49

Post by dwaltrip »

deadbabe wrote: post #12
Dylan16807 wrote: post #6

Earlier quoted context omitted.

Models are improving every day. People are figuring out thousands of different optimizations to training and to hardware efficiency. The idea that right now in early June 2025 is when improvement stops beggars belief. We might be approaching a limit, but that's going to be a sigmoid curve, not a sudden halt in advancement.

5 years ago a person would be blown away by today’s LLMs. But people today will merely say “cool” at whatever LLMs are in use 5 years from now. Or maybe not even that.

Bold prediction…

Re: I read all of Cloudflare's Claude-generated commits

#50

Post by BoorishBears »

sitkack wrote: post #19
Dylan16807 wrote: post #6

Earlier quoted context omitted.

Models are improving every day. People are figuring out thousands of different optimizations to training and to hardware efficiency. The idea that right now in early June 2025 is when improvement stops beggars belief. We might be approaching a limit, but that's going to be a sigmoid curve, not a sudden halt in advancement.

It is copium that it will suddenly stop and the world they knew before will return. ChatGPT came out in Nov 2022. Attention Was All There Was in 2017, we were already 5 years in the past. Or 5 years of research to catch up to, and then from 2022 to now ... papers and research have been increasing exponentially. Even in if SOTA models were frozen, we still have years of research to apply and optimize in various ways.

I think it's equally copium that people keep assuming we're just going to compound our way into intelligence that generalizes enough to stop us from handholding the AI, as much as I'd genuinely enjoy that future.

Lately I spend all day post-training models for my product, and I want to say 99% of the research specific to LLMs doesn't reproduce and/or matter once you actually dig in.

We're getting exponentially more papers on the topics and they're getting worse on average.

Every day there's a new paper claiming an X% gain by post-training some ancient 8B parameter model and comparing it to a bunch of other ancient models after they've overfitted on the public dataset of a given benchmark and given the model a best of 5.

And benchmarks won't ever show it, but even ChatGPT 3.5-Turbo has better general world knowledge than a lot models people consider "frontier" models today because post-training makes it easy to cover up those gaps with very impressive one-prompt outputs and strong benchmark scores.

-

It feels like things are getting stuck in a local maxima: we are making forward progress, the models are useful and getting more useful, but the future people are envisioning takes reaching a completely different goal post that I'm not at all convinced we're making exponential progress towards.

There maybe exponential number of techniques claiming to be ground breaking, but what has actually unlocked new capabilities that can't just as easily be attributed to how much more focused post-training has become on coding and math?

Test time compute feels like the only one and we're already seeing the cracks form in terms of its effect on hallucinations, and there's a clear ceiling for the performance the current iteration unlocks as all these models are converging on pretty similar performance after just a few model releases.

Post reply on HN