71–80 of 82 posts
Each time a new LLM version comes out, I give it another try at generating tests. However, even with the latest models, tailored GPTs, and well-crafted prompts with code examples, the same issues keep surfacing: - The models often create several tests within the same equivalence class, which barely expands test coverage - They either skip parameterization, creating multiple redundant tests, or go overboard with 5+ pa…
Earlier quoted context omitted.
It's a toy. It doesn't do useful work. The code is fine for the pathetically small sample but that coding style does not scale to real software scales.
|_____| ... ... ... |_____| I guess we all see where the goalposts will be tomorrow. Good code, good problem, I don't like the language. Or something :)
Earlier quoted context omitted.
I believe that's called mutation testing. Using an LLM to perform the mutation sounds like a great idea
LLMs are not suitable for mutation testing. Mutation testing needs to be fast to be useful (because you need to generate and test a lot of mutated versions); an LLM-based mutator would be extremely slow as well as error-prone.
Also you only need to generate a set of mutations for any particular unit once, and then again when the test code or the code under test changes.
I actually tested Claude Sonnet to see how it would fare at writing a test suite for a background worker. My previous experience was with some version of GPT via Copilot, and it was... not good. I was, however, extremely impressed with Claude this time around. Not only did it do a great job off the bat, but it taught me some techniques and tricks available in the language/framework (Ruby, Rspec) which I wasn't famili…
Despite some of the comments on this thread, despite it not wanting to be true, I must admit LLM's are impressive. Software engineers and ML specialists have finally invented the thing which disrupts their own jobs substantially either via large reduction in hours and/or reduction in staff. As the hours a software engineer spends coding diminishes by large factors so too especially in this economy will hours spent required paying an engineer will fall up to the point where anyone can create code and learn from an LLM as you have just done. Once everybody is special, no one is and fundamentally employment, and value of things created from software, comes from scarcity just like everything else in our current system.
I think there's probably only a few years left where software engineers are around - or at least seen as a large part of an organization with large teams, etc. Yes AI software will have bugs, and yes it won't be perfect but you can get away with just one or two for a whole org to fix the odd blip of an LLM. It feels like people are picking on minor things at this point, which while true, for a business those costs are "meh" while the gains of removing engineers are substantial.
I want to be wrong; but every time I see someone "learning from LLM's", saving lots of time doing stuff, saving 100's of hours, etc I think its only 2-3 years in and already its come this far.
I actually tested Claude Sonnet to see how it would fare at writing a test suite for a background worker. My previous experience was with some version of GPT via Copilot, and it was... not good. I was, however, extremely impressed with Claude this time around. Not only did it do a great job off the bat, but it taught me some techniques and tricks available in the language/framework (Ruby, Rspec) which I wasn't famili…
All this makes me think making software engineers redundant is really the "killer app" of LLM's. This is where the AI labs are spending most of the effort - its the best marketing after all for their product - fear sells better than greed (loss aversion) making engineers notice and unable to dismiss it. Despite some of the comments on this thread, despite it not wanting to be true, I must admit LLM's are impressive.…
Maybe. A lot of places have headcount limits on software devs because of budget constraints. As in, the reason they don't hire more is because they can't afford it, not because there is a shortage of code to write and bugs to give. The more optimistic view is that the nature of being a software engineer will adjust to increased productivity and focus on the parts of the job that LLMs can't do, with the market for experts who are skilled at removing "the odd blip from an LLM". Expertise will also move into areas where there's less or insufficient training data for a particular niche. One way to future proof yourself is to find places where it frequently makes up non existent libraries and is bad at code in a language, and specialize in that.
Earlier quoted context omitted.
|_____| ... ... ... |_____| I guess we all see where the goalposts will be tomorrow. Good code, good problem, I don't like the language. Or something :)
"Dreck" means worthless rubbish. Code that solves useless toy problems is worthless rubbish.
Take SQLite Wasm as an example:
https://simonwillison.net/2024/Oct/21/claude-artifacts/#sqli...
Perhaps you don’t use database everyday, but a web based sql client is very common, very far from worthless, let alone rubbish. Imagine a developer being able to stand up this module in 5 mins, using it as the starting point for further work.
Earlier quoted context omitted.
> Using an autocomplete to "bang it out" seems foolish. Based on my own experience, I find the widespread scepticism on HN about AI-assisted coding misplaced. There will be corner cases, there will be errors, and there will be bugs. There will also be apps for which AI is not helpful at all. But that's fine - nobody is saying otherwise. The question is only about whether it is a _significant_ nett saving on the time…
Ok but looking at those tests for just a second (for createElement), you might want to go through it again, or ask the computer or whatever. For example, edgeCases.test.ts is totally redundant, you are running the same exact tests in children.test.ts. Edit: such a LLM repo... why did it feel the need to recreate these DOM types? Is your AI just trying to maximize LoC? It just seems like such a pain and potential sour…
But generally, the tests are very useful. My point is that there will be redundancies, and maybe even bugs - and that's fine, because the time needed to fix these would be much less than what it would have taken to write them from scratch.
Earlier quoted context omitted.
LLMs are not suitable for mutation testing. Mutation testing needs to be fast to be useful (because you need to generate and test a lot of mutated versions); an LLM-based mutator would be extremely slow as well as error-prone.
Set aside LLMs, why does mutation testing need to be fast? It would be fine to have mutation tests run slowly, out-of-band of the main CI pipeline. They aren't mission critical, they're smoke tests for your unit tests. Also you only need to generate a set of mutations for any particular unit once, and then again when the test code or the code under test changes.
Earlier quoted context omitted.
"Dreck" means worthless rubbish. Code that solves useless toy problems is worthless rubbish.
If you are to share your code with us, are you sure that we wouldn’t find any worthless rubbish code? If they do exist, can you be certain that they took you less than 5 mins to build like the GP asserted? Take SQLite Wasm as an example: https://simonwillison.net/2024/Oct/21/claude-artifacts/#sqli... Perhaps you don’t use database everyday, but a web based sql client is very common, very far from worthless, let alone…