Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I read all of Cloudflare's Claude-generated commits

maxemitchell.com

Post reply on HN
Print view View original

221–230 of 291 posts

Re: I read all of Cloudflare's Claude-generated commits

#221

Post by potholereseller »

gizmo686 wrote: post #59
SrslyJosh wrote: post #5

> Reading through these commits sparked an idea: what if we treated prompts as the actual source code? Imagine version control systems where you commit the prompts used to generate features rather than the resulting implementation. Please god, no, never do this. For one thing, why would you not commit the generated source code when storage is essentially free? That seems insane for multiple reasons. > When models ine…

My work has involved a project that is almost entirely generated code for over a decade. Not AI generated, the actual work of the project is in creating the code generator. One of the things we learned very quickly was that having generated source code in the same repository as actual source code was not sustainable. The nature of reviewing changes is just too different between them. Another thing we learned very qui…

> The end result of that was that we had to significantly rearchitect the project for us to essentially inject manually crafted code into arbitrary places in the generated code.

This sounds like putting assembly in C code. What was the input language? These two bits ("Not AI generated", "a feature flag") suggest that the code generator didn't have a natural language frontend, but rather a real programming language frontend.

Did you or anyone else inform management that a code generator is essentially a compiler with extra characters? [0] If yes, then what was their response?

I am concerned that your current/past work might have been to build a Compiler-as-a-Service (CaaS). [1] No shade, I'm just concerned that other managers might read all this and then try to build their own CaaS.

[0] Yes, I'm implying that LLMs are compilers. Altman has played us for fools; he's taught a billion people the worst part of programming: fighting the compiler to give you the output you want.

[1] Compiler-as-a-Service is the future our forefathers couldn't imagine warning us about. LLMs are CaaS's; time is a flat circle; where's the exit?; I want off this ride.

Re: I read all of Cloudflare's Claude-generated commits

#222

Post by layer8 »

tptacek wrote: post #204
buu700 wrote: post #202

Earlier quoted context omitted.

Because if the LLM is a black box and its output must ultimately be verified by humans, then you can't treat conversion of prompts into code as a simple build step as though an AI agent were just some sort of compiler. You still need to persist the actual code in source control. (I assume that isn't what you're actually arguing against, in which case at least one of us must have misread something from the parent chai…

Right, you definitely can't do that. People do talk as if the question was whether we could stick LLM calls into Makefiles. Nobody would ever do that, at least not with the technology we have at hand.

But that’s exactly what the part of the article quoted in the root comment (https://news.ycombinator.com/item?id=44206758) is about, and in reference to future AI technology. That’s was this subthread was discussing.

Re: I read all of Cloudflare's Claude-generated commits

#223

Post by layer8 »

diggan wrote: post #82
mschild wrote: post #61

Earlier quoted context omitted.

> One of the things we learned very quickly was that having generated source code in the same repository as actual source code was not sustainable. Keeping a repository with the prompts, or other commands separate is fine, but not committing the generated code at all I find questionable at best.

If you can 100% reproduce the same generated code from the same prompts, even 5 years later, given the same versions and everything then I'd say "Sure, go ahead and don't saved the generated code, we can always regenerate it". As someone who spent some time in frontend development, we've been doing it like that for a long time with (MB+) generated code, keeping it in scm just isn't feasible long-term. But given this…

Temperature > 0 isn’t a problem as long as you can specify/save the random seed and everything else is deterministic. Of course, “as long as” is still a tall order here.

Re: I read all of Cloudflare's Claude-generated commits

#224

Post by gizmo686 »

potholereseller wrote: post #221
gizmo686 wrote: post #59

Earlier quoted context omitted.

My work has involved a project that is almost entirely generated code for over a decade. Not AI generated, the actual work of the project is in creating the code generator. One of the things we learned very quickly was that having generated source code in the same repository as actual source code was not sustainable. The nature of reviewing changes is just too different between them. Another thing we learned very qui…

> The end result of that was that we had to significantly rearchitect the project for us to essentially inject manually crafted code into arbitrary places in the generated code. This sounds like putting assembly in C code. What was the input language? These two bits ("Not AI generated", "a feature flag") suggest that the code generator didn't have a natural language frontend, but rather a real programming language fr…

The input was a highly structured pdf specification of a family of protocols and formats. Essentially, a real language with very stupid parsing requirements and the occasional typo. The PDF itself was clearly intended for human consumption, but I'm 99% sure that someone somewhere at some point had a machine readable specification that was used to generate most of the PDF. Sadly, no one seems to know where to even start looking for such a thing.

> Did you or anyone else inform management that a code generator is essentially a compiler with extra characters?

The output of the code generator was itself fed into a compiler that we also built; and about half of the codegen team (myself included) were themselves developers for the compiler.

I think management is still scared from the 20 year old M4 monstrosity we are still maintaining because writing a compiler would be "too complex".

Re: I read all of Cloudflare's Claude-generated commits

#225

Post by layer8 »

handoflixue wrote: post #220
lowsong wrote: post #213

Earlier quoted context omitted.

I'm the first to admit that I'm an AI skeptic, but this goes way beyond my views about AI and is a fundamentally unsound idea. Let's assume that a hypothetical future AI is perfect. It will produce correct output 100% of the time, with no bugs, errors, omissions, security flaws, or other failings. It will also generate output instantly and cost nothing to run. Even with such perfection this idea is doomed to failure…

We already have AI agents that can ask a human for help / clarification in those cases. It could also analyze the company website, marketing materials, and so forth, and use that to infer the missing pieces. (Again, something that exists today)

If the AI has to ask for clarification, you can’t run it as a reproducible build step as envisaged. It’s as if your compiler would pause to ask clarifying questions on each CI run.

If the company website, marketing materials, and so forth become part of the input, you’ll have to put those in version control as well, as any change is likely to result in a different application being generated (which may or may not be what you want).

Re: I read all of Cloudflare's Claude-generated commits

#226

Post by layer8 »

fastball wrote: post #30
SrslyJosh wrote: post #5

> Reading through these commits sparked an idea: what if we treated prompts as the actual source code? Imagine version control systems where you commit the prompts used to generate features rather than the resulting implementation. Please god, no, never do this. For one thing, why would you not commit the generated source code when storage is essentially free? That seems insane for multiple reasons. > When models ine…

The idea as stated is a poor one, but a slight reshuffling and it seems promising: You generate code with LLMs. You write tests for this code, either using LLMs or on your own. You of course commit your actual code: it is required to actually run the program, after all. However you also save the entire prompt chain somewhere. Then (as stated in the article), when a much better model comes along, you re-run that chain…

One reason we treat tests that way is that we don’t generally rewrite the application from scratch, but usually only refactor parts of the existing code or make smaller changes. If we regularly did the former, test suites would have to be much mire comprehensive than they typically are. Not to mention that the tests need to change when the API changes, so you generally have to rewrite the unit tests along with the application and can’t apply them unchanged.

Re: I read all of Cloudflare's Claude-generated commits

#227

Post by auggierose »

kiitos wrote: post #197
auggierose wrote: post #77

Earlier quoted context omitted.

Exactly my thought. This is just natural language as a specification language.

...as an ambiguous and inadequately-specified specification language.

In the end, every specification is specified via natural language, this is just where the buck stops. All math books are written in natural language, even the ones about specification languages.

Re: I read all of Cloudflare's Claude-generated commits

#228

Post by layer8 »

auggierose wrote: post #78
fch42 wrote: post #67

Earlier quoted context omitted.

It won't make it impossible for junior engineers to learn. It will simply reduce the amount of opportunities to learn (and not just for juniors), by virtue of companies' beancounters concluding "two for one" (several juniors) doesn't return the same as "buy one get one free" (existing staff + AI license). I dread the day we all "learn from AI". The social interaction part of learning is just as important as the conte…

I learnt programming on my own, without any social interaction involved. In fact, I loved programming because it does not involve any social interaction. Programming has become more of a "social game" in the last 15 years or so. AI is a new superpower for people like me, bringing balance to the Force.

To me, LLMs are just a different kind of social interaction I mostly don’t want, tedious and frustrating.

Re: I read all of Cloudflare's Claude-generated commits

#229

Post by auggierose »

layer8 wrote: post #228
auggierose wrote: post #78

Earlier quoted context omitted.

I learnt programming on my own, without any social interaction involved. In fact, I loved programming because it does not involve any social interaction. Programming has become more of a "social game" in the last 15 years or so. AI is a new superpower for people like me, bringing balance to the Force.

To me, LLMs are just a different kind of social interaction I mostly don’t want, tedious and frustrating.

But it is not a social interaction. An LLM is a machine.

I think there is also a big difference between being forced to use an LLM in a certain way, and being able to completely design your interaction with the LLM yourself. The former I imagine can be indeed tedious and frustrating, the latter is just miraculous.

Re: I read all of Cloudflare's Claude-generated commits

#230

Post by layer8 »

maxemitchell wrote: post #146
thorum wrote: post #23

Humorous that this article has a strong AI writing smell - the author should publish the prompts they used!

I did human notes -> had Claude condense and edit -> manually edit. A few of the sentences (like the stinky one below) were from Claude which I kept if it matched my own thoughts, though most were changed for style/prose. I'm still experimenting with it. I find it can't match style at all, and even with the manual editing it still "smells like AI" as you picked up. But, it also saves time. My prompt was essentially "…

I wonder if those notes wouldn’t have been more interesting as-is, and possibly also more condensed.
Post reply on HN