Live data from Hacker News

I read all of Cloudflare's Claude-generated commits

maxemitchell.com

31–40 of 291 posts

Re: I read all of Cloudflare's Claude-generated commits

#31
I used almost 100% AI to build a SCUMM-like parser, interpreter, and engine (https://github.com/fpgaminer/scumm-rust). It was a fun workflow; I could generally focus on my usual work and just pop in occasionally to check on and direct the AI.

I used a combination of OpenAI's online Codex, and Claude Sonnet 4 in VSCode agent mode. It was nice that Codex was more automated and had an environment it could work in, but its thought-logs are terrible. Iteration was also slow because it takes awhile for it to spin the environment up. And while you _can_ have multiple requests running at once, it usually doesn't make sense for a single, somewhat small project.

Sonnet 4's thoughts were much more coherent, and it was fun to watch it work and figure out problems. But there's something broken in VSCode right now that makes its ability to read console output inconsistent, which made things difficult.

The biggest issue I ran into is that both are set up to seek out and read only small parts of the code. While they're generally good at getting enough context, it does cause some degradation in quality. A frequent issue was replication of CSS styling between the Rust side of things (which creates all of the HTML elements) and the style.css side of things. Like it would be working on the Rust code and forget to check style.css, so it would just manually insert styles on the Rust side even though those elements were already styled on the style.css side.

Codex is also _terrible_ at formatting and will frequently muck things up, so it's mandatory to use it with an autoformatter and instructions to use it. Even with that, Codex will often say that it ran it, but didn't actually run it (or ran it somewhere in the middle instead of at the end) so its pull requests fail CI. Sonnet never seemed to have this issue and just used the prevailing style it saw in the files.

Now, when I say "almost 100% AI", it's maybe 99% because I did have to step in and do some edits myself for things that both failed at. In particular neither can see the actual game running, so they'd make weird mistakes with the design. (Yes, Sonnet in VS Code can see attached images, and potentially can see the DOM of vscode's built in browser, but the vision of all SOTA models is ass so it's effectively useless). I also stepped in once to do one major refactor. The AIs had decided on a very strange, messy, and buggy interpreter implementation at first.

Re: I read all of Cloudflare's Claude-generated commits

#32

I asked this in the other thread (no response, but I was a bit late) How does anyone using AI like this have confidence that they aren't unintentionally plagiarizing code and violating the terms of whatever license it was released under? For random personal projects I don't see it mattering that much. But if a large corp is releasing code like this, one would hope they've done some due diligence that they have to jus…

As an individual dev, I simply don’t care. Not my problem.

Companies are satisfied with the idemnity provided by Microsoft.

Re: I read all of Cloudflare's Claude-generated commits

#33
post #11

> Reading through these commits sparked an idea: what if we treated prompts as the actual source code? Imagine version control systems where you commit the prompts used to generate features rather than the resulting implementation. Please god, no, never do this. For one thing, why would you not commit the generated source code when storage is essentially free? That seems insane for multiple reasons. > When models ine…

>> what if we treated prompts as the actual source code? You would not do this because: unlike programming languages, natural languages are ambiguous and thus inadequate to fully specify software.

Prompts are like story on the board, and like engineers, depends on the understanding of the model the generated source code can vary. Saying the prompts could be the actual code is so wrong and dangerous thought

Re: I read all of Cloudflare's Claude-generated commits

#34

> Prompts as Source Code Another way to phrase this is LLM-as-compiler and Python (or whatever) as an intermediate compiler artefact. Finally, a true 6th generation programming language! I've considered building a toy of this with really aggressive modularisation of the output code (eg. python) and a query-based caching system so that each module of code output only changes when the relevant part of the prompt or ups…

> Finally, a true 6th generation programming language!

Karpathy already said English is the new programming language.

Re: I read all of Cloudflare's Claude-generated commits

#35

It's an interesting review but I really dislike this type of techno-utopian determinism: "When models inevitably improve..." Says who? How is it inevitable? What if they've actually reached their limits by now?

More compute mean more faster processing, more context.

Re: I read all of Cloudflare's Claude-generated commits

#36

I asked this in the other thread (no response, but I was a bit late) How does anyone using AI like this have confidence that they aren't unintentionally plagiarizing code and violating the terms of whatever license it was released under? For random personal projects I don't see it mattering that much. But if a large corp is releasing code like this, one would hope they've done some due diligence that they have to jus…

> How does anyone using AI like this have confidence that they aren't unintentionally plagiarizing code and violating the terms of whatever license it was released under?

They don’t and no one cares

Re: I read all of Cloudflare's Claude-generated commits

#37

> Reading through these commits sparked an idea: what if we treated prompts as the actual source code? Imagine version control systems where you commit the prompts used to generate features rather than the resulting implementation. Please god, no, never do this. For one thing, why would you not commit the generated source code when storage is essentially free? That seems insane for multiple reasons. > When models ine…

The idea as stated is a poor one, but a slight reshuffling and it seems promising: You generate code with LLMs. You write tests for this code, either using LLMs or on your own. You of course commit your actual code: it is required to actually run the program, after all. However you also save the entire prompt chain somewhere. Then (as stated in the article), when a much better model comes along, you re-run that chain…

Means the temperature should be set to 0 (which not every provider supports) so that the output becomes entirely deterministic. Right now with most models if you give the same input prompt twice it will give two different solutions.

Re: I read all of Cloudflare's Claude-generated commits

#38

I asked this in the other thread (no response, but I was a bit late) How does anyone using AI like this have confidence that they aren't unintentionally plagiarizing code and violating the terms of whatever license it was released under? For random personal projects I don't see it mattering that much. But if a large corp is releasing code like this, one would hope they've done some due diligence that they have to jus…

This is an excellent question that the AI-boosters always seem to dance around. Three replies already are saying “Nobody cares.” Until they do. I’d be willing to bet that some time in the near future, some big company is going to care a lot and that there will be a landmark lawsuit that significantly changes the LLM landscape. Regulation or a judge is going to eventually decide the extent to which someone can use AI to copy someone else’s IP, and it’s not going to be pretty.

Re: I read all of Cloudflare's Claude-generated commits

#39

> Reading through these commits sparked an idea: what if we treated prompts as the actual source code? Imagine version control systems where you commit the prompts used to generate features rather than the resulting implementation. Please god, no, never do this. For one thing, why would you not commit the generated source code when storage is essentially free? That seems insane for multiple reasons. > When models ine…

It’s been a thing people have done for at least a year https://github.com/i365dev/LetterDrop

Re: I read all of Cloudflare's Claude-generated commits

#40

Earlier quoted context omitted.

The idea as stated is a poor one, but a slight reshuffling and it seems promising: You generate code with LLMs. You write tests for this code, either using LLMs or on your own. You of course commit your actual code: it is required to actually run the program, after all. However you also save the entire prompt chain somewhere. Then (as stated in the article), when a much better model comes along, you re-run that chain…

Means the temperature should be set to 0 (which not every provider supports) so that the output becomes entirely deterministic. Right now with most models if you give the same input prompt twice it will give two different solutions.

Even at temp 0, you might get different answers, depending on your inference engine. There might be hardware differences, as well as software issues (e.g. vLLM documents this, if you're using batching, you might get different answers depending on where in the batch sequence your query landed).
Post reply on HN