AI handles incidents, engineers lose touch with their systems • Hacker News Board

Re: AI handles incidents, engineers lose touch with their systems

#211

Post by spockz » Sat, Sep 05, 2026, 1:06 PM UTC

orwin wrote: post #185

spockz wrote: post #135

Earlier quoted context omitted.

Exactly. People ask how we get seniors with juniors using llm. The answer is the same. Review the code. Analyse write down what is wrong. What you expect to have been better. Force every change to be documented and explained enough.

It just takes forever now. The understanding is lower, the effort is lower, and frankly, I think the interest is lower too. I might be in the last generation who truly had fun working on a 'shrodinger' bug.

Yes. So find someone with intrinsic desire to engineer. Mentor them.

In the mean time put a plethora of guardrails in place to make sure the AI Train doesn’t derail production.

Oh. And keep showing your value. In the end every org can do with less low paid overeager uninterested juniors. Might as well let agents Do those tasks.

Re: AI handles incidents, engineers lose touch with their systems

#212

Post by backlava12 » Sat, Sep 05, 2026, 1:06 PM UTC

Richbeach wrote: post #150

willtemperley wrote: post #129

Earlier quoted context omitted.

> 30 minutes of patient, old school troubleshooting This is also the best way to understand a codebase, and it's quite enjoyable. I think the models are getting too egotistical. They're so confident of their fixes they won't bother suggesting basic techniques like isolation of the problem by disabling code paths, for example. They'll keep shotgunning less-and-less likely fixes with undiminishing confidence until the…

> and it's quite enjoyable Used to be somewhat enjoyable. Nothing pleasant about digging around codebase that was heavily affected by the last 12-18 months of AI-ing.

[dead]

Re: AI handles incidents, engineers lose touch with their systems

#213

Post by backlava12 » Sat, Sep 05, 2026, 1:08 PM UTC

neuronic wrote: post #117

konschubert wrote: post #61

Earlier quoted context omitted.

It's the uncanny valley of AI. It's still not quite good enough yet that you can trust it blindly on a big codebase, so you still have to read and understand everything - which is often harder than just writing it up yourself. EDIT: don’t get me wrong. I still think AI is incredibly useful for a lot of tasks! But when implementing an architecturally hairy thing, I find it less stressful and equally quick to jump down…

Your assumption is that LLMs will ever leave this uncanny valley. Maybe unforeseen breakthroughs and different architectures are achieved. Given LLM fundamental shortcomings grounded in mathematics and information theory, I highly doubt they will and we will always need to deal with these issues in some capacity.

[dead]

Re: AI handles incidents, engineers lose touch with their systems

#214

Post by intended » Sat, Sep 05, 2026, 1:08 PM UTC

gnfargbl wrote: post #130

intended wrote: post #121

Earlier quoted context omitted.

The article proposes a resolution based on the challenge and evidence they (correctly) identify at the start. The simulation solution is backed by their own applied example, and is also the logical intervention based on how the problem is set up. Perhaps Bainbridge is doing more heavy lifting for me, by filling in blanks and gaps that others are finding.

I would say that Bainbridge supports my position, if anything. Take a look at section 2.3 of her paper, where she lists all the reasons why simulation can be unsuitable in preparing for complex incidents: https://davidjusth.com/s/Ironies-of-Automation_Bainbridge_19...

Thats an interesting subsection, and I wouldn't say it agrees fully with either position. It both acknowledges the option of using simulations, and the weaknesses inherent in it being unable to simulate unknown unknowns.

Re: AI handles incidents, engineers lose touch with their systems

#215

Post by shakna » Sat, Sep 05, 2026, 1:08 PM UTC

KronisLV wrote: post #162

shakna wrote: post #95

Earlier quoted context omitted.

Its different, in that when you teach that engineer, they either leave because now they hate you, or they grow. They change to meet the standards of a project, rather than inventing their own. We don't get seniors, without juniors. I'd say more than half the job, is just... Learning. People grow.

> Its different, in that when you teach that engineer, they either leave because now they hate you, or they grow. Or they just have their own hubris and ignore your (provably better) suggestions because their way is "better/easier/how we've always done things". And then you end up with someone sprinkling N+1 issues throughout the system and making systems with bad architectures throughout the years, not thinking abou…

I have never had it be every now and then. It is always bullshit, the first time around. Usually followed by, "You ignored the first three rules, and all the examples, try again."

I have worked with some amazingly incompetent devs. Some promoted into place to become someone else's problem, and some parachuted in through connections, and never once have they brought the continuous and unevolving frustration of modern LLMs.

I taught one of our "React SME"s, what Typescript even was. And they, were less likely to throw a ten thousand line fix, where twenty would do. They didn't see that the DB had a list of validation rules, and copy and paste those into seven different files, instead of just querying the DB, so the code would be kept up to date. And whilst I might have had to repeat that the DD is our source of truth, they never argued that implementing the DD wasn't aligned to the intended design.

AI isn't forcing people to be better with knowledge transfer. It's taking conversations that happened in person, that were back and forth and gave both people a better understanding of a thing, and turning it into a half-assed and out-of-date wiki. Without all the things that make a wiki actually useful to onboard the next newcomer.

I do get what some of these frustrations are, and where they come from. I'm the official documentation maintainer, because nobody else ever wrote anything down on expectations. Its fairly thankless. But... All I've seen AI do in that same realm, is exacerbate misunderstanding.

Like when it presented an Apex script (Salesforce) guaranteed to exceed governor limits to our junior, who took it and ran with it. And when it aborted, the AI spent half a day leading the junior around and telling them to change config settings - that the environment was the problem, not the code.

The fix? Toss the three layers of Queueables, and use Database.insert on a list. Job done. PR for 150LoC, instead of the offered 780LoC.

(As a complete aside, can anyone convince Salesforce to fix the documentation for: https://developer.salesforce.com/docs/atlas.en-us.apexcode.m...)

Re: AI handles incidents, engineers lose touch with their systems

#216

Post by CoastalCoder » Sat, Sep 05, 2026, 1:11 PM UTC

bob1029 wrote: post #6

A natural evolution of engineers losing touch with the customers and users. I'm noticing some of the concern play out regarding AI weakening the capabilities of software people. I gave the team an exact solution on a silver platter and they still failed to identify how to go about it after 3 days slamming it into Claude. The resolution is literally 1 line of code that could be arrived at in about 30 minutes of patien…

I think I've heard late-career engineers from the past century say similar things, although I can't remember exactly quotes.

I wonder if every generation of engineers ends up with such hand-wringing about the next.

I'm also genuinely curious what fraction of the time they were right to worry.

Re: AI handles incidents, engineers lose touch with their systems

#217

Post by backlava12 » Sat, Sep 05, 2026, 1:11 PM UTC

Tade0 wrote: post #113

danielheath wrote: post #65

Earlier quoted context omitted.

I'm seeing people who _used to be_ like that losing that understanding without realizing it's happening - they have a superficial idea of what the code is doing, enough to feel like they understand it, but the change is apparent when watching them handle something unexpected.

LLMs reduced interest on tech debt, but it's still there and people who have a tendency to acquire it will go bankrupt eventually.

[dead]

Re: AI handles incidents, engineers lose touch with their systems

#218

Post by podocarp » Sat, Sep 05, 2026, 1:11 PM UTC

itsalwaysgood wrote: post #174

konschubert wrote: post #146

Earlier quoted context omitted.

I think my point is that its sometimes (!) easier to use the manual chisel rather than go for the automatic chisel and then fix its mistakes.

They say writing engages more of the brain and helps us to remember what's written more than if we just read it, or copy and paste. When you say it's easier to go manual, it seems you're talking about learning retention. And you're right. But seniors have learned enough that they're able to iterate quickly with AI. They know how to organize their work, manage change, tasks. They know how to break a problem down into…

It's not just learning is it, why do people buy hand ground coffee when there quite literally isn't any difference? Or audiophile snake oil? As long as humans are still the consumers, some part of consumption will be emotional. Could be to support local artisans, could be gullibility, could be love, whatever.

Maybe one day artisanal code will be a thing lol. Hand written like calligraphy. Those with refined tastes will have their favorite code artisans. And the plebs can continue with mass produced industrial junk.

Re: AI handles incidents, engineers lose touch with their systems

#219

Post by seer » Sat, Sep 05, 2026, 1:12 PM UTC

solatic wrote: post #112

Author has a good head on their shoulders, but few if any companies are going to spend time on incident simulations for their SREs. Why not? Because even pre-AI, very few companies spend time practicing restoring their backups, or disaster recovery, or picking infrequently-used runbooks to practice, or seeing whether they can easily rotate secrets without downtime, or trying to redploy the system onto another vendor'…

Hmm that’s strange, in my experience it is the other way around - Claude is super diligent with infra and will _insist_ on double checking and trying everything for real before committing.

When I was doing this myself I would read the docs and just implement them - claud is going about doing real software archeology to figure if what is said is actually the truth or it’s stale/inaccurate/buggy.

I’ve become 10 times more diligent because it is a lot easier to do. It’s no longer Urgh it’s good enough let’s ship it, now it’s “sure put a leg on it to figure it out and double check it”.

Backups are _tested regularly_ now because LLMs make it cheap to do so.

The only problem is when new engineers who haven’t learned these things Pre-ai now don’t really get why it is needed in the first place and will often lead the agent astray.

I think to address this we need to change or improve our training routines in general for humans. I think a lot of companies nowadays just skip that and deploy a company wide skill/policy for the agents, but don’t transfer the underlying skills to the devs themselves.

Re: AI handles incidents, engineers lose touch with their systems

#220

Post by msandford » Sat, Sep 05, 2026, 1:13 PM UTC

This is the same problem as the "AI drives the car until it can't" situation. Feel free to doze off so long as you can wake up and instantly have world class racecar "save the situation" reflexes. Hope isn't a strategy but that's what all of this feels like.