41–50 of 82 posts
I did this for Laravel a few months ago and it’s great. It’s basically the same as the article describes, and it has definitely increased the number of tests I write. Happy to open source if anyone is interested.
I am very sceptical of LLM (or any AI) code generation usefulness and it does not really have anything to do with AI itself. In the past I've been involved in several projects deeply using MDA (Model Driven Architecture) techniques which used various code generation methods to develop software. One of the main obstacles was the problem of maintaining the generated code. IOW: how should we treat generated code? If we…
You should NEVER modify generated code. All of our generated code is pretended with a big comment that says "GENERATED CODE DO NOT MODIFY. This code could be regenerated at any time and any changes will be lost." If you need to change behaviour of generated code you need to change your generator to provide the right hooks. Obviously none of this applies to "AI" generated code because the "AI" generator is not determi…
In every single system I have worked on, tests were not just tests - they were their own parallel application, and it required careful architecture and constant refactoring in order for it to not get out of hand. "More tests" is not the goal - you need to write high impact tests, you need to think about how to test the most of your app surface with least amount of test code. Sometimes I spend more time on the test co…
Based on my own experience, I find the widespread scepticism on HN about AI-assisted coding misplaced. There will be corner cases, there will be errors, and there will be bugs. There will also be apps for which AI is not helpful at all. But that's fine - nobody is saying otherwise. The question is only about whether it is a _significant_ nett saving on the time spent across various project types. The answer to that is a resounding Yes.
The entire set of tests for a web framework I wrote recently were generated with Claude and GPT. You can see them here: https://github.com/webjsx/webjsx/tree/main/src/test
On an average, these tests are better than tests I would have written myself. The project was written mostly by AI as well, like most other stuff I've written since GPT4 came out.
"Using an autocomplete to bang it out" is exactly what one should do - in most cases.
Earlier quoted context omitted.
> We don't expect human developers to be perfect, why should we expect AI assistants. What absolute nonsense. What an absurd false equivalence. It's not that we expect perfection or even human level performance from "AI". It's that the crap that comes out of LLMs is not even at the level of a first year student. I've never in my entire life reviewed the code of a junior engineer and seen them invent third party APIs…
That problem genuinely doesn't matter to me at all. If an LLM hallucinates a method that doesn't exist I find out the moment I try and run the code. If I'm using ChatGPT Code Interpreter (for Python) or Claude analysis mode (for JavaScript) I don't even have to intervene: the LLM can run in a loop, generating code, testing that it executes without errors and correcting any mistakes it makes. I still need to carefully…
Earlier quoted context omitted.
That problem genuinely doesn't matter to me at all. If an LLM hallucinates a method that doesn't exist I find out the moment I try and run the code. If I'm using ChatGPT Code Interpreter (for Python) or Claude analysis mode (for JavaScript) I don't even have to intervene: the LLM can run in a loop, generating code, testing that it executes without errors and correcting any mistakes it makes. I still need to carefully…
Yes I've seen the dreck you produce with LLMs. Not a shining endorsement in my eyes. https://news.ycombinator.com/item?id=41929174
I think the source code for tools like this one is genuinely good code: https://github.com/simonw/tools/blob/main/extract-urls.html
What do you see that's wrong with that?
In every single system I have worked on, tests were not just tests - they were their own parallel application, and it required careful architecture and constant refactoring in order for it to not get out of hand. "More tests" is not the goal - you need to write high impact tests, you need to think about how to test the most of your app surface with least amount of test code. Sometimes I spend more time on the test co…
Mostly agree. My first thought when I read this post was: Is his goal to test the code, or validate the features? The first problem is he's providing the code, and asking for tests. If his code has a bug, the tests will enshrine those bugs. It's like me writing some code, and then giving it to a junior colleague, not providing any context, and saying "Hey, write some tests for this." This is backwards. I'm not a TDD…
You can provide the context to an AI model though, you can share the source with it.
In every single system I have worked on, tests were not just tests - they were their own parallel application, and it required careful architecture and constant refactoring in order for it to not get out of hand. "More tests" is not the goal - you need to write high impact tests, you need to think about how to test the most of your app surface with least amount of test code. Sometimes I spend more time on the test co…
Mostly agree. My first thought when I read this post was: Is his goal to test the code, or validate the features? The first problem is he's providing the code, and asking for tests. If his code has a bug, the tests will enshrine those bugs. It's like me writing some code, and then giving it to a junior colleague, not providing any context, and saying "Hey, write some tests for this." This is backwards. I'm not a TDD…
Adding tests that capture the current state of things, so that when that bug is uncovered tests can easily be updated to the correct functionality to prove the bug prior to fixing it is a much better place to be than the status quo.
The horse may have bolted from the barn, but we can at least close the farm gate in the hopes of recapturing it eventually.
Earlier quoted context omitted.
Yes I've seen the dreck you produce with LLMs. Not a shining endorsement in my eyes. https://news.ycombinator.com/item?id=41929174
Which of those did you think were dreck? I think the source code for tools like this one is genuinely good code: https://github.com/simonw/tools/blob/main/extract-urls.html What do you see that's wrong with that?
In every single system I have worked on, tests were not just tests - they were their own parallel application, and it required careful architecture and constant refactoring in order for it to not get out of hand. "More tests" is not the goal - you need to write high impact tests, you need to think about how to test the most of your app surface with least amount of test code. Sometimes I spend more time on the test co…
This seems like the kind of thing that should be highly dependent on the kind of project one is doing, if you have an MVP and your test code is taking longer than the actual code then it is clear the test code is antagonistic to the whole concept of an MVP.