Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

An AI Vibe Coding Horror Story

tobru.ch

Post reply on HN
Print view View original

211–220 of 224 posts

Re: An AI Vibe Coding Horror Story

#211

Post by animuchan »

monkeydust wrote: post #65
delis-thumbs-7e wrote: post #4

Meanwhile on Linkedin… Every sales bozo with zero technical understanding is screaming top of their virtual lungs that evrything must be done with AI and it is solution to every layoff, economic problem, everything. It is just a matter of time when something really really bad happens.

Looks like bad stuff is happening, really bad is a bit scary if you qualify that as threat to life or livelihood. Let's see what the next generation of models bring to this equation.

The next generation of the user is what I think will be the factor for AI-assisted loss of life, moreso than the models themselves. It's the trust we put in them.

The dinosaurs of the before-LLM age will increasingly struggle to convince young people e.g. to seek competent medical consultation, because ChatGPT can do that much better (it's always supportive, available at any hour, meets you where you are...) -- this aspect alone is bound to have a death toll associated with it.

Or imagine asking ChatGPT to map an Alpine climbing route, -- anything with a bad failure mode.

Re: An AI Vibe Coding Horror Story

#212

Post by animuchan »

nerptastic wrote: post #134
shivaniShimpi_ wrote: post #111

Earlier quoted context omitted.

interesting, so the ai got the hard stuff right. password hashing, schema design, fine. it fumbled the stuff that isn't really "coding" knowledge, feels more like an operational intuition? backup folder sitting in web root isn't a security question, it's a "have you ever been burned before" question, and surgeon hadn't. so they didn't ask and the model didn't cover it, imo that's the actual pattern. the model secures…

This is what I’m noticing. At my workplace, we have 3 or 4 non-devs “writing” code. One was trying to integrate their application with the UPS API. They got the application right, and began stumbling with the integration - created a developer account, got the API key, but in place of the applications URL, the had input “localhost:5345” and couldn’t get that to work, so they gave up. They never asked the tech team wha…

What I see in the workplace is, people specifically outsource decisions to LLM. It tries to flag and explain all sorts of landmines, it really sometimes does, but the prompt is "make it work" and "be relentless", and the operator is barely even looking at the (conversational) output of the LLM, just the code (or other file) they asked for.

This is another difference to a largely organic developer: the ability to refuse a massively damaging or stupid task.

Re: An AI Vibe Coding Horror Story

#213

Post by grey-area »

edwinjm wrote: post #19
grey-area wrote: post #12

Earlier quoted context omitted.

This problem is unrelated to CI and dev practices etc, this is about trusting the output of generative AI without reading it, then using it to handle patient data. Vibe coding is just a bad idea, unless you’re willing and able to vet the output, which most people doing it are not.

It is related to CI and dev practices etc. A experienced developer using AI would add security/data protection, even when vibe coding.

An experienced developer would not have created this mess nor 'vibe-coded' (i.e. used AI without checking), but this person probably didn't know what they didn't know and believed the AI when it confidently asserted this mess was the correct way to do this.

None of that is related to the practice of Continuous Integration.

Re: An AI Vibe Coding Horror Story

#214

Post by t43562 »

sajithdilshan wrote: post #206
t43562 wrote: post #196

Earlier quoted context omitted.

Management know how to keep themselves safe! They watch your AI usage and pressure you to use more. They tell you that "of course you must check the code" but of course if you do then they can start telling you your performance isn't good enough. "Fred is much faster than you" and of course Fred says he checks everything but does he really? To do code reviews he just uses an AI anyhow so nobody catches what the AI do…

You do realize that bad management would still be bad management regardless of AI right? "Fred is much faster than you" has been used by management way before AI was there will aways be scenarios like that regardless of AI or any other tool or technology. Blaming bad management on advances on AI is quite naive and it actually doesn't achieve anything. If the management is the problem then you have to address that and…

Oh definitely! :-) I just think it's like an amplification of the same old thing. It makes it easier to play that game and harder to counter it.

Fred now generates 5,000 lines of horse-dung that appears to work and management are gob-smacked. It is extremely fragile, has no security and the tests are all autogenerated so nobody knows if they're even testing what is actually important but...

Above the team-lead level, management, product manager etc have no idea what's inside a piece of work that makes it maintainable or secure or anything else and all they see is their idea realised and the person who did it has a golden halo so you cannot say a single negative thing about the work without a tonne of shit pouring on you.

This has happened to me. It was in the days when ChatGPT was much worse than it is now and the code was almost one big hallucination - indescribable how bad it was. The only advantage I had was that the whole team, other than Fred of course, rejected the PR. It caused a world of horrible problems though and incredible behavior from "Fred" and yet he was able to get away with it until he finally stepped so far over the line that nobody could support him. It caused other team members to leave though so it was a disaster.

Re: An AI Vibe Coding Horror Story

#215

Post by ZephyrBlu »

consumer451 wrote: post #202
ZephyrBlu wrote: post #167

Earlier quoted context omitted.

People like to make this point, but traditional engineering has the opposite problem: insanely overwrought processes and box-checking that exists for no reason and slows everything down to a snail's pace. Yes there are safety-critical parts, but they surrounded by a ton of bullshit. It's also absurd to think that there is no company which does genuine software "engineering". If you break ads at Google/Meta, streaming…

That's quite the take. Throughout human history there were lots of instances of vibe-engineering and vibe-architecting, in the physical world. Since the failings of not doing proper engineering is far more evident, the reasons for the "insanely overwrought processes and box-checking that exists for no reason and slows everything down to a snail's pace" go back to the earliest written law, AKA the Code of Hammurabi, c…

No point in discussing with someone who is arguing in bad faith. I already agreed that some parts of the engineering process are safety critical. If you think there is no bullshit in the process you don't have enough knowledge about the requirements imposed by e.g. building regulations.

Re: An AI Vibe Coding Horror Story

#216

Post by sajithdilshan »

t43562 wrote: post #214
sajithdilshan wrote: post #206

Earlier quoted context omitted.

You do realize that bad management would still be bad management regardless of AI right? "Fred is much faster than you" has been used by management way before AI was there will aways be scenarios like that regardless of AI or any other tool or technology. Blaming bad management on advances on AI is quite naive and it actually doesn't achieve anything. If the management is the problem then you have to address that and…

Oh definitely! :-) I just think it's like an amplification of the same old thing. It makes it easier to play that game and harder to counter it. Fred now generates 5,000 lines of horse-dung that appears to work and management are gob-smacked. It is extremely fragile, has no security and the tests are all autogenerated so nobody knows if they're even testing what is actually important but... Above the team-lead level,…

Your case sounds pretty extreme, like a combination of multiple toxic factors and I still don't think it is due to AI. Fred still could have pushed bad code without the help of AI and the situation would be the same. In this case the problem is Fred, not AI. It's the human negligence of Fred that caused the issue. Even if you ban using AI in your company, still Fred would behave the same and find a different way to be toxic.

Re: An AI Vibe Coding Horror Story

#217

Post by [deleted] »

Ekaros wrote: post #177
shivaniShimpi_ wrote: post #125

Every other field that's figured out high stakes failure models eventually landed on the same solution - make sure two people that understand the details are looking at it - pilots have copilots surgeons with checklists and nuclear plants have independent verification. Software was always the exception, cause when it broke it mostly just broke for you, vibe coding is not going to change the equation, it barely remove…

We do have code reviews for pull requests. But on average I would guess there is great amount of complacency there. I suppose old proper QA phase was best answer we had. But that is expensive and slow.

[deleted]

Re: An AI Vibe Coding Horror Story

#218

Post by shivaniShimpi_ »

NoGravitas wrote: post #193
Ekaros wrote: post #177

Earlier quoted context omitted.

We do have code reviews for pull requests. But on average I would guess there is great amount of complacency there. I suppose old proper QA phase was best answer we had. But that is expensive and slow.

Maybe expensive and slow is actually an improvement.

i mean i don't disagree with you, but then there's at least unit test cases that we write as devs in a certain way that ai just doesn't understand quite well yet. we worked so much on speed that we forgot along the way that slow was a safety net. we tend to let our qa teams leverage ai as much as they want to automate test cases but i don't feel super comfortable getting rid of it just yet

Re: An AI Vibe Coding Horror Story

#219

Post by shivaniShimpi_ »

TeMPOraL wrote: post #155
shivaniShimpi_ wrote: post #111

Earlier quoted context omitted.

interesting, so the ai got the hard stuff right. password hashing, schema design, fine. it fumbled the stuff that isn't really "coding" knowledge, feels more like an operational intuition? backup folder sitting in web root isn't a security question, it's a "have you ever been burned before" question, and surgeon hadn't. so they didn't ask and the model didn't cover it, imo that's the actual pattern. the model secures…

Maybe this is what's missing in the prompt? We've learned years ago to tell the AI they're the expert principal 100x software developer ninja, but maybe we should also honestly disclose our own level of expertise in the task. A simple "I'm a professional surgeon, but sadly know nothing about making software" would definitely make the conversation play out differently. How? Needs to be seen. But in an idealized scenar…

this is more on the lines of the marketing play that "developers are out of job" or "oh i created an app overnight", well sure you did but you also put yourself at a lot of risk, a lot of ai tools are just so risky at times, i'm tbh surprised why claude still asks for api keys to be provided in terminal. you could almost bet that the devs building it 100% know they should not do it, but there's always a way you can brainstorm with the ai, crosscheck with stackoverflow, reddit or anything like you did earlier. this is no substitute for the way programming worked, just a 100% faster and efficient engine for sure. it's such an under-explored area that there's def more eyes needed on it

Re: An AI Vibe Coding Horror Story

#220

Post by shivaniShimpi_ »

NoGravitas wrote: post #192
shivaniShimpi_ wrote: post #111

Earlier quoted context omitted.

interesting, so the ai got the hard stuff right. password hashing, schema design, fine. it fumbled the stuff that isn't really "coding" knowledge, feels more like an operational intuition? backup folder sitting in web root isn't a security question, it's a "have you ever been burned before" question, and surgeon hadn't. so they didn't ask and the model didn't cover it, imo that's the actual pattern. the model secures…

The competence profile of any LLM-based AI is extremely spiky - whether it does a particular task well or not is pretty independent of the (subjective) difficulty of the task. This is very different from our experience with humans.

slow was the safety net for sure but then there were errors too, there's a sweet equilibrium spot where ai + human oversight reaches that efficient + almost perfect situation. ofc with the right methods
Post reply on HN