Earlier quoted context omitted.
>the failure mode is invisible Only if you are missing tests for what counts for you. And that's true for both dev-written code, and for vibed code.
Who writes the tests?
[dead]
Breaking the spell of vibe coding
261–270 of 375 posts
Re: Breaking the spell of vibe coding
#262Earlier quoted context omitted.
It doesn't seem obvious that it's a problem for LLM coders to write their own tests (if we assume that their coding/testing abilities are up to snuff), given human coders do so routinely.
This thread is talking about vibe coding, not LLM-assisted human coding. The defining feature of vibe coding is that the human prompter doesn't know or care what the actual code looks like. They don't even try to understand it. You might instruct the LLM to add test cases, and even tell it what behavior to test. And it will very likely add something that passes, but you have to take the LLM's word that it properly te…
> This thread is talking about vibe coding, not LLM-assisted human coding.
I was writing about vibe-coding. It seems these guys are vibe-coding (https://factory.strongdm.ai/) and their LLM coders write the tests.
I've seen this in action, though to dubious results: the coding (sub)agent writes tests, runs them (they fail), writes the implementation, runs tests (repeat this step and last until tests pass), then says it's done. Next, the reviewer agent looks at everything and says "this is bad and stupid and won't work, fix all of these things", and the coding agent tries again with the reviewer's feedback in mind.
Models are getting good enough that this seems to "compound correctness", per the post I linked. It is reasonable to think this is going somewhere. The hard parts seem to be specification and creativity.
Re: Breaking the spell of vibe coding
#263Earlier quoted context omitted.
> Vibe coding would be catastrophic here. Not because the AI can't write the code - it usually can - but because the failure mode is invisible. A hallucinated edge case in a tax calculation doesn't throw an error. It just produces a slightly wrong number that gets posted to a real accounting platform and nobody notices until the accountant does their review. How is that different from handwritten code ? Sounds like s…
It’s shocking to me that people even ask this type of question. How do you not see the difference between a machine that will hallucinate something random if it doesn’t know the answer vs a human that will logic through things and find the correct answer.
I know right, had this discussion last week and it's difficult to argue when people are blinded by the "magic" and hype of the slot machine.
Re: Breaking the spell of vibe coding
#264> However, it is important to ask if you want to stop investing in your own skills because of a speculative prediction made by an AI researcher or tech CEO. I don't think these are exclusive. Almost a year ago, I wrote a blog post about this [0]. I spent the time since then both learning better software design and learning to vibe code. I've worked through Domain-Driven Design Distilled , Domain-Driven Design , Imple…
I personally found out that knowing how to use ai coding assistants productively is a skill like any other and a) it requires a significant investment of time b) can be quite rewarding to learn just as any other skill c) might be useful now or in the future and d) doesn't negate the usefulness of any other skills acquired on the past nor diminishes the usefulness of learning new skills in the future
As much as i loved the relation of vibe coding to slots and their related flow states in this article, I also think what you are stating is the exact reason these tools are not the same as slots, the skill gap is there and its massive.
I think there are a ton of people just pulling the lever over and over, instead of stepping back and considering how they should pull the lever. When you step back and consider this, you are for sure going to end up falling deeper into the engineering, architecture realm. Ensuring that continually pulling the lever doesn't result in potential future headaches.
I think a ton of people in this community are struggling with the lose of flow state, and attempting to still somehow enter it through prompting. The game in my view has just changed, its more about just generating the code, and being thoughtful about what comes next, its rapid usage of a junior to design your system, and if you overdue the rapidness the junior will give you headaches.
Re: Breaking the spell of vibe coding
#265Earlier quoted context omitted.
It doesn't seem obvious that it's a problem for LLM coders to write their own tests (if we assume that their coding/testing abilities are up to snuff), given human coders do so routinely.
Maybe it’s just the people I’m around but assuming you write good tests is a big assumption. It’s very easy to just test what you know works. It’s the human version of context collapse, becoming myopic around just what you’re doing in the moment, so I’d expect LLMs to suffer from it as well.
> the human version of context collapse, becoming myopic around just what you’re doing in the moment
The setups I've seen use subagents to handle coding and review, separately from each other and from the "parent" agent which is tasked with implementing the thing. The parent agent just hands a task off to a coding agent whose only purpose is to do the task, the review agent reviews and goes back and forth with the coding agent until the review agent is satisfied. Coding agents don't seem likely to suffer from this particular failure mode.
Re: Breaking the spell of vibe coding
#266[dead]
Re: Breaking the spell of vibe coding
#267Earlier quoted context omitted.
Assembly will not help you further up the stack which is working with agents, not writing code (obsolete skill). Apparently my /s was needed
I got your poor attempt at sarcasm. I just don't think it's a good argument. The person who understands how lower levels of abstraction work, will always run circles technically around those who don't. Besides, "AI" tools are not a higher level of abstraction, and can't be compared to compilers. Their goal is to replace all cognitive work done by humans. If you think programming is obsolete, the same will eventually…
And I got your cheeky, dismissive attitude which completely misses the forest for the trees.
> In the meantime, programmers will be in demand to fix issues caused by vibe coders.
Yes, I agree. They’ll be lower on the totem pole than the vibe coders, too. Because the best vibe coders have the same skill set as you - years of classical engineering background. So how can one differentiate themself in the new world? I aspire to move up the totem pole, not down, and leaning into AI is the #1 way to do that. Staying a “bug fixer” only is what will push you out of employment.
Re: Breaking the spell of vibe coding
#268Earlier quoted context omitted.
It’s shocking to me that people even ask this type of question. How do you not see the difference between a machine that will hallucinate something random if it doesn’t know the answer vs a human that will logic through things and find the correct answer.
I know right, had this discussion last week and it's difficult to argue when people are blinded by the "magic" and hype of the slot machine.
Which interestingly is the meat of this article. The key points aren’t that “vibe coding is bad” but that the design and experience of these tools is actively blinding and seductive in a way that impairs ability to judge effectiveness.
Re: Breaking the spell of vibe coding
#269Earlier quoted context omitted.
> Vibe coding would be catastrophic here. Not because the AI can't write the code - it usually can - but because the failure mode is invisible. A hallucinated edge case in a tax calculation doesn't throw an error. It just produces a slightly wrong number that gets posted to a real accounting platform and nobody notices until the accountant does their review. How is that different from handwritten code ? Sounds like s…
It’s shocking to me that people even ask this type of question. How do you not see the difference between a machine that will hallucinate something random if it doesn’t know the answer vs a human that will logic through things and find the correct answer.
It's shocking to me that people make this claim as if humans, especially in some legacy accounting system, would somehow be much better at (1) recognizing their mistakes, and (2) even when they don't, not fudge-fingering their implementation. Like the criticisms of agents are valid, but the incredulity that they will ever be used in production or high risk systems to me is just as incredible. Of course they will -- where is Opus 4.6 compared to Sonnet 4? We've hit an inflection point where replacing hand coding with an agent and interacting only via prompt is not only doable, highly skilled people are already routinely doing it. Companies are already _requiring_ that people do it. We will then hit an inflection point at some time soon where the incredulity at using agents even in the highest stakes application will age really really poorly. Let's see!
Re: Breaking the spell of vibe coding
#270Earlier quoted context omitted.
It doesn't seem obvious that it's a problem for LLM coders to write their own tests (if we assume that their coding/testing abilities are up to snuff), given human coders do so routinely.
This thread is talking about vibe coding, not LLM-assisted human coding. The defining feature of vibe coding is that the human prompter doesn't know or care what the actual code looks like. They don't even try to understand it. You might instruct the LLM to add test cases, and even tell it what behavior to test. And it will very likely add something that passes, but you have to take the LLM's word that it properly te…
The issue I have with using LLM's is the test code review. Often the LLM will make a 30 or 40 line change to the application code. I can easily review and comprehend this. Then I have to look at the 400 lines of generated test code. While it may be easy to understand there's a lot of it. Go through this cycle several times a day and I'm not convinced I'm doing a good review of the test code do to mental fatigue, who knows what I may be missing in the tests six hours into the work day?