Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I read all of Cloudflare's Claude-generated commits

maxemitchell.com

Post reply on HN
Print view View original

281–290 of 291 posts

Re: I read all of Cloudflare's Claude-generated commits

#281

Post by latexr »

isaacremuant wrote: post #245
latexr wrote: post #87

Earlier quoted context omitted.

Can we please stop using the em-dash as a metric to “detect” LLM writing? It’s lazy and wrong. Plenty of people use em-dashes, it’s a useful punctuation mark . If humans didn’t use them, they wouldn’t be in the LLM training data. There are better clues, like the kind of vague pretentious babble bad marketers use to make their products and ideas seem more profound than they are. It’s a type of bad writing which looks…

It's not lazy and wrong. It's a fantastic indicator. > If humans didn’t use them, they wouldn’t be in the LLM training data. Humans weren't using them in every context as they are now. They might've been used in books but blog posts and work documents weren't full of them. It's not a definite thing but it's absolutely a good indicator.

Blog posts, news articles, and other web texts have been using correct punctuation marks for a long time. I know because I’ve been noticing misuses (usually having switched or repeated characters for quotes) for over a decade.

Plenty of people care about typographic punctuation, and others use software (such as Apple’s OSs, markdown converters, publishing and editing tools) which auto-converts smart punctuation. Heck, tools for doing that are older than Markdown, and that is already two decades old.

https://daringfireball.net/projects/smartypants/

Look, nowhere have I said using an em-dash can’t be an indicator, my objection is people using it as the indicator. It’s become a meme. Too many people act like if the existence of a single em-dash immediately and conclusively proves it was written by an LLM. It does not.

Re: I read all of Cloudflare's Claude-generated commits

#282

Post by auggierose »

kiitos wrote: post #271
auggierose wrote: post #269

Earlier quoted context omitted.

How is ABNF itself specified? Yes, via natural language. And the Go language spec is written in natural language, too, you can check for yourself: https://go.dev/ref/spec

ABNF itself is specified with a well-defined grammar and syntax...

Yes, but the spec is still done in natural language: https://www.rfc-editor.org/rfc/rfc5234

Just like any other RFC.

Re: I read all of Cloudflare's Claude-generated commits

#283

Post by kentonv »

59nadir wrote: post #280
kentonv wrote: post #273

Earlier quoted context omitted.

> The code that any current-gen LLM generates, no matter how precise the prompt it's given, is never even close to the quality standards expected of any senior-level engineer, in any organization I've been a part of, at any point in my career. You are just making assertions here with no evidence. If you prompt the LLM for code, and then you review the code, identify specific problems, and direct the LLM to fix those…

It's possible kiitos has (or had?) a higher standard in mind for what should constitute a senior/"lead engineer" at Cloudflare and how much they should be constrained by typing as part of implementation. Out of interest: How much did the entire process take and how much would you estimate it to take without the LLM in the loop?

> It's possible kiitos has (or had?) a higher standard in mind for what should constitute a senior/"lead engineer" at Cloudflare and how much they should be constrained by typing as part of implementation.

See again here, you're implying that I or my code is disappointing somehow, but with no explanation for how except that it was LLM-assisted. I assert that the code is basically as good as if I'd written it by hand, and if you think I'm just not a competent engineer, like, feel free to Google me.

It's not the typing itself that constrains, it's the detailed but non-essential decision-making. Every line of code requires making several decisions, like naming variables, deciding basic structure, etc. Many of these fine-grained decisions are obvious or don't matter, but it's still mentally taxing, which is why nobody can write code as fast as they can type even when the code is straightforward. LLMs can basically fill in a bunch of those details for you, and reviewing the decisions -- especially the fine-grained ones that don't matter -- is a lot faster than making them.

> How much did the entire process take and how much would you estimate it to take without the LLM in the loop?

I spent about five days mostly focused on prompting the LLM (although I always have many things interrupting me throughout the day, so I wasn't 100% focused). I estimate it would have taken me 2x-5x as long to do by hand, but it's of course hard to say for sure.

Re: I read all of Cloudflare's Claude-generated commits

#284

Post by kiitos »

kentonv wrote: post #273
kiitos wrote: post #267

Earlier quoted context omitted.

Forget about reliably-correct. The code that any current-gen LLM generates, no matter how precise the prompt it's given, is never even close to the quality standards expected of any senior-level engineer, in any organization I've been a part of, at any point in my career. They very much never produce code that is as good as what I can create. If the LLM-generated code you're seeing passes this level of muster, in you…

> The code that any current-gen LLM generates, no matter how precise the prompt it's given, is never even close to the quality standards expected of any senior-level engineer, in any organization I've been a part of, at any point in my career. You are just making assertions here with no evidence. If you prompt the LLM for code, and then you review the code, identify specific problems, and direct the LLM to fix those…

> If you prompt the LLM for code, and then you review the code, identify specific problems, and direct the LLM to fix those problems, and repeat, you can, in fact, end up with production-ready code

I guess I will concede that this is possible, yes. I've never seen it happen, myself, but it could be the case, at some point, in the future.

> in less time than it would take to write by hand.

This is my point of contention. The process you've described takes ages longer than however much time it would take a competent senior-level engineer to just type the code from first principles. No meaningful project has ever been bottle-necked on how long it takes to type characters into editors.

All of that aside, the claim you're making here is that, speaking as a senior IC, the code that an LLM produces, guided by your prompt inputs, is more or less equivalent to any code that you could produce yourself, even controlling for time spent. Which just doesn't match any of my experiences with any current-gen LLM or agent or workflow or whatever. If your universe is all about glue code, where typing is enemy no. 1, and details don't matter, then fair enough, but please understand that this is not usually the domain of senior-level engineers.

Re: I read all of Cloudflare's Claude-generated commits

#285

Post by tptacek »

kiitos wrote: post #177
simonw wrote: post #124

Earlier quoted context omitted.

You seem to be using "hallucinate" to mean "makes mistakes". That's not how I use it. I see hallucination as a very specific kind of mistake: one where the LLM outputs something that is entirely fabricated, like a class method that doesn't exist. The agent compiler/linter loop can entirely eradicate those. That doesn't mean the LLM won't make plenty of other mistakes that don't qualify as hallucinations by the defini…

I'm not using "hallucinate" to mean "makes mistakes". I'm using it to mean "code that is syntactically correct and passes tests but is semantically incoherent". Which is the same thing that "hallucination" normally means in the context of a typical user LLM chat session.

Why would you merge code that was "semantically incoherent"? And how does the answer to that question, about "hallucinations" that matter in practice, allow you to then distinguish between "hallucinations" and "bugs"?

Re: I read all of Cloudflare's Claude-generated commits

#286

Post by simonw »

kiitos wrote: post #284
kentonv wrote: post #273

Earlier quoted context omitted.

> The code that any current-gen LLM generates, no matter how precise the prompt it's given, is never even close to the quality standards expected of any senior-level engineer, in any organization I've been a part of, at any point in my career. You are just making assertions here with no evidence. If you prompt the LLM for code, and then you review the code, identify specific problems, and direct the LLM to fix those…

> If you prompt the LLM for code, and then you review the code, identify specific problems, and direct the LLM to fix those problems, and repeat, you can, in fact, end up with production-ready code I guess I will concede that this is possible, yes. I've never seen it happen, myself, but it could be the case, at some point, in the future. > in less time than it would take to write by hand. This is my point of contenti…

"the code that an LLM produces, guided by your prompt inputs, is more or less equivalent to any code that you could produce yourself, even controlling for time spent"

That's been my personal experience over the past 1.5 years. LLMs, prompted and guided by me, write code that I would be proud to produce without them.

Re: I read all of Cloudflare's Claude-generated commits

#287

Post by rossjudson »

tptacek wrote: post #121
kiitos wrote: post #115

Earlier quoted context omitted.

The sequence of commits talked about by the OP -- i.e. kenton's coding session's commits -- are like one degree removed from 100% pure vibe coding.

Your claim here being that Kenton Varda isn't reading the code he's generating. Got it. Good note.

You ever get the feeling someone didn't look up Kenton Varda before criticizing the code he's generating?

I guarantee you that Kenton Varda's generators generate more code than any other code generators that aren't compilers. ;)

Re: I read all of Cloudflare's Claude-generated commits

#288

Post by kentonv »

kiitos wrote: post #284
kentonv wrote: post #273

Earlier quoted context omitted.

> The code that any current-gen LLM generates, no matter how precise the prompt it's given, is never even close to the quality standards expected of any senior-level engineer, in any organization I've been a part of, at any point in my career. You are just making assertions here with no evidence. If you prompt the LLM for code, and then you review the code, identify specific problems, and direct the LLM to fix those…

> If you prompt the LLM for code, and then you review the code, identify specific problems, and direct the LLM to fix those problems, and repeat, you can, in fact, end up with production-ready code I guess I will concede that this is possible, yes. I've never seen it happen, myself, but it could be the case, at some point, in the future. > in less time than it would take to write by hand. This is my point of contenti…

I have only claimed that for this particular project it worked really well, and was much faster than writing by hand. This particular project was arguably a best-case scenario: a greenfield project implementing a well-known standard against a well-specified design.

I have tried using AI to make changes to the Cloudflare Workers Runtime -- my usual main project, which I started, and know like the back of my hand, and which incidentally handles over a trillion web requests every day -- and in general in that case I haven't found it saved me much time. (Though I've been a bit surprised by the fact that it can find its way around the code at all, it's a pretty complicated C++ codebase.)

It really depends on the use case.

Re: I read all of Cloudflare's Claude-generated commits

#289

Post by 59nadir »

kentonv wrote: post #283
59nadir wrote: post #280

Earlier quoted context omitted.

It's possible kiitos has (or had?) a higher standard in mind for what should constitute a senior/"lead engineer" at Cloudflare and how much they should be constrained by typing as part of implementation. Out of interest: How much did the entire process take and how much would you estimate it to take without the LLM in the loop?

> It's possible kiitos has (or had?) a higher standard in mind for what should constitute a senior/"lead engineer" at Cloudflare and how much they should be constrained by typing as part of implementation. See again here, you're implying that I or my code is disappointing somehow, but with no explanation for how except that it was LLM-assisted. I assert that the code is basically as good as if I'd written it by hand,…

> See again here, you're implying that I or my code is disappointing somehow, but with no explanation for how except that it was LLM-assisted. I assert that the code is basically as good as if I'd written it by hand, and if you think I'm just not a competent engineer, like, feel free to Google me.

I think you're reading a bit too deeply into what I wrote; I explained what I interpreted kiitos' posts as essentially saying. I realize that you've probably had to deal with a lot of people being skeptical to the point of "throwing shade", as it were, so I understand the defensive posture. I am skeptical, but the reason I'm asking questions (alongside the previous bit) is because I'm actually curious about your experiment.

> It's not the typing itself that constrains, it's the detailed but non-essential decision-making. Every line of code requires making several decisions, like naming variables, deciding basic structure, etc. Many of these fine-grained decisions are obvious or don't matter, but it's still mentally taxing, which is why nobody can write code as fast as they can type even when the code is straightforward. LLMs can basically fill in a bunch of those details for you, and reviewing the decisions -- especially the fine-grained ones that don't matter -- is a lot faster than making them.

In your estimation, what is your mental code coverage of the code you ended up with? Do you feel like you have a complete mapping of it, i.e. you could get an external request for change and map it quickly to where it needs to be made and why exactly there?

Re: I read all of Cloudflare's Claude-generated commits

#290

Post by kentonv »

59nadir wrote: post #289
kentonv wrote: post #283

Earlier quoted context omitted.

> It's possible kiitos has (or had?) a higher standard in mind for what should constitute a senior/"lead engineer" at Cloudflare and how much they should be constrained by typing as part of implementation. See again here, you're implying that I or my code is disappointing somehow, but with no explanation for how except that it was LLM-assisted. I assert that the code is basically as good as if I'd written it by hand,…

> See again here, you're implying that I or my code is disappointing somehow, but with no explanation for how except that it was LLM-assisted. I assert that the code is basically as good as if I'd written it by hand, and if you think I'm just not a competent engineer, like, feel free to Google me. I think you're reading a bit too deeply into what I wrote; I explained what I interpreted kiitos' posts as essentially sa…

> In your estimation, what is your mental code coverage of the code you ended up with? Do you feel like you have a complete mapping of it, i.e. you could get an external request for change and map it quickly to where it needs to be made and why exactly there?

I know the code structure about as well as if I had written it.

Honestly the code structure is not very complicated. It flows pretty naturally from the interface spec in the readme, and I'd expect anyone who knows OAuth could find their way around pretty easily.

But yes, as part of prompting improvements to the code, I had to fully understand the implementation. My prompts are entirely based on reading the code and deciding what needed to be changed -- not based on any sort of black-box testing of the code (which would be "vibe coding").

Post reply on HN