AI handles incidents, engineers lose touch with their systems • Hacker News Board

Re: AI handles incidents, engineers lose touch with their systems

#361

Post by mjevans » Sun, Sep 06, 2026, 2:48 AM UTC

LeafItAlone wrote: post #204

Richbeach wrote: post #150

Earlier quoted context omitted.

> and it's quite enjoyable Used to be somewhat enjoyable. Nothing pleasant about digging around codebase that was heavily affected by the last 12-18 months of AI-ing.

Depends on the prior state. For one codebase I work in, LLM written code is noticeably and measurably (we have literally measured it for bugs, performance, etc.) better than what existed before. So the LLM code is a welcome change.

I've only personally utilized it for code _review_ and always ignored the suggested result code instead to understand the source of the issue and try to arrive at a resolution I find satisfactory for my mental model of the issue and solutions I'd prefer.

My take is that the main difference is the approach to problems the coding agents have. They optimize towards presuming a fully working invocation environment, yet checking everything anyway, and then rolling back any changes and re-testing that such changes worked. Which is great for normal types of software but incredibly tedious for anything aimed at less than a fully hands free automated environment.

In other words, AI is very awesome at scope creep of assigned problems and targeting the validation prompts baked into the review system.

Re: AI handles incidents, engineers lose touch with their systems

#362

Post by fastest963 » Sun, Sep 06, 2026, 3:27 AM UTC

alexpotato wrote: post #347

solatic wrote: post #112

Author has a good head on their shoulders, but few if any companies are going to spend time on incident simulations for their SREs. Why not? Because even pre-AI, very few companies spend time practicing restoring their backups, or disaster recovery, or picking infrequently-used runbooks to practice, or seeing whether they can easily rotate secrets without downtime, or trying to redploy the system onto another vendor'…

I was the head of a global SRE team at a top tier hedge fund. We started doing a weekly meeting where whoever was on call would do a table top exercise of an outage that happened the prior week. The idea was to have someone else be the simulated person on call while the SRE from last week's oncall would talk them through the symptoms, what happened and where to look. The idea was to spread knowledge around how incide…

How often are there outages? Is there a significant outage every week to do simulations for? The problem with our organization is that outages are so rare that weekly we have nothing to discuss.

Re: AI handles incidents, engineers lose touch with their systems

#363

Post by sharts » Sun, Sep 06, 2026, 3:46 AM UTC

As societies evolve people lose touch with nature —growing food, hunting, building shelter, and making clothing.

Progress usually means adding levels of abstraction so that more can be done with less effort and needing to worry about lower levels.

AI isn’t all that different.

Re: AI handles incidents, engineers lose touch with their systems

#364

Post by andrekandre » Sun, Sep 06, 2026, 3:48 AM UTC

sillyfluke wrote: post #182

itsalwaysgood wrote: post #134

Earlier quoted context omitted.

I'd say it's more about learning how to organize your work more efficiently. If you think about a product like marble: it's something that most be chiseled out of time. Some people can chisel better products: the AI is just a better chisel. Sometime still has to guide the chisel and judge the art/product. In our cases, the market judges products.

This seems to misunderstand the parent comment. The sibling comment plays along with the analogy and mentions having to fix the mistakes of an automatic chisel, but the original comment is alluding to the fact that it is definitely not an automatic chisel. There is no uncanny valley in an automatic chisel. Programming up until this point was done by using deterministic tools to build products. LLMs appear to be nonde…

  > But as this type of task is literally the least liked task in programming, the programmer passes it off to the LLM as well to avoid burnout...

this is my daily life now and its miserable

its an overload spiral that gets worse at each iteration (even while 'gardening' our agents/skills)

the worst part is using the llm to generate the specs/features; its shit from the start and tumbles downhill from there...

Re: AI handles incidents, engineers lose touch with their systems

#365

Post by andrekandre » Sun, Sep 06, 2026, 4:06 AM UTC

anothereng wrote: post #299

krtkush wrote: post #9

I find the use of AI like quicksand. The more I use it, the more I have to rely on it to make changes/ fix things in the same system. In the end, I come out feeling empty; no intuitive knowledge of the system "I" built or fixed. Code review is important but it does not replace the mental model I am able to build when I do all the steps of software development manually without AI.

I put on AGENTS.MD a prompt that says: teach me something small something about the changes you did if the changes were < 3 tokens if it was more than that teach me more. That's helped me somwhat. So now everytime it changes something it teaches me something about the codebase

i literally tell it "explain it like a 5 year old can understand" and "keep it under 100 characters".. helps somewhat (its still an llm so caveat emptor)

Re: AI handles incidents, engineers lose touch with their systems

#366

Post by lazyasciiart » Sun, Sep 06, 2026, 4:26 AM UTC

bob1029 wrote: post #6

A natural evolution of engineers losing touch with the customers and users. I'm noticing some of the concern play out regarding AI weakening the capabilities of software people. I gave the team an exact solution on a silver platter and they still failed to identify how to go about it after 3 days slamming it into Claude. The resolution is literally 1 line of code that could be arrived at in about 30 minutes of patien…

But was this team capable of patient, old school troubleshooting before they used AI? I’ve seen someone paid as a senior engineer define the root cause of an outage as “that code was written before our team started here”. (I spent half an hour arguing with him and then ended the RCA).

Re: AI handles incidents, engineers lose touch with their systems

#367

Post by nharziro » Sun, Sep 06, 2026, 4:37 AM UTC

bigstrat2003 wrote: post #241

eudamoniac wrote: post #237

Earlier quoted context omitted.

I find this shocking though not unbelievable. Could you share how you measure this accurately? I'm interested in integrating such measurements into our services as well.

I find it unbelievable. I've seen the code LLMs write and it sucks compared to what a typical human produces. The only way an LLM is doing better than human programmers is if your human programmers were producing really terrible work.

What exactly do you mean when you say llm generated code? Are people prompting llms for changes and features without reviewing the code or iterating on it and then comparing that to what human writes? Because if so it's not surprising that you're getting worse results. Humans also write code through iteration. You can definitely get llms to write good code by enforcing guardrails and constraints through tooling and agent.md, and iterative reviews to nudge towards what you want. The first pass will look nothing like the committed code. I don't expect the llm to one shot anything.

Re: AI handles incidents, engineers lose touch with their systems

#368

Post by glouwbug » Sun, Sep 06, 2026, 4:54 AM UTC

thunky wrote: post #356

glouwbug wrote: post #348

Earlier quoted context omitted.

Major critical software has always needed thousands of engineers to maintain million line codebases. LLMs can't reason across edge case failure modes, and better yet, LLMs generate compartmentalized modules which exponentially explodes the number of failure modes as LLM based codebases grow. Read how everyone here laments about Sol or Opus or GPT6 generates 10x the required number of lines

> LLMs generate compartmentalized modules which exponentially explodes the number of failure modes It sounds like you're arguing against modularizing which is an unusual stance. Especially for a million line codebase.

https://wiki.c2.com/?RavioliCode

Re: AI handles incidents, engineers lose touch with their systems

#369

Post by franciscop » Sun, Sep 06, 2026, 5:26 AM UTC

scruple wrote: post #232

Richbeach wrote: post #150

Earlier quoted context omitted.

> and it's quite enjoyable Used to be somewhat enjoyable. Nothing pleasant about digging around codebase that was heavily affected by the last 12-18 months of AI-ing.

We have a tool at work that was entirely spun out of a prototype that itself was entirely driven by a single guy with an unlimited Claude budget. I only touch it on the edges, where I need to integrate it every so slightly into one of my systems. It's a crucial piece of software for the team, though. I've looked over the source, the thing that strikes me the most is the verbosity. Human software engineers would've be…

It really depends on the codebase. I remember ~10 years ago trying to learn about some of the big codebases that existed and what struke me was the verbosity. I don't remember if it was a PHP framework or a big JS library, but it was like 20+ lines of comment/explanation for each line of code. I was trying to follow the code flow and it was basically impossible, you could fit like 2-5 lines of actual code per screen.

Re: AI handles incidents, engineers lose touch with their systems

#370

Post by deadbabe » Sun, Sep 06, 2026, 6:27 AM UTC

saltcured wrote: post #289

deadbabe wrote: post #262

Earlier quoted context omitted.

Nothing wrong with compartmentalization. Some people like it, some people don’t. Just different ways to live your life.

I don't think "compartmentalization" does this topic justice. Note, I'm not accusing you, since I don't know your work. My post is about the general milieu and the rhetoric around it. This "compartment" terminology seems to whitewash a dimension that has diligence on one end, and fraud on the other. I've always worked in high-trust organizations where we depended on each other operating on the diligent side of things…

This view is a little idealistic.

Software Engineers have historically not been very liable even for errors they create by their own hands, and companies tend to operate on a “blameless” engineering culture where there is not really a formal consequence for screwing something up. Your peers may not trust you for a bit, but the trust builds back rapidly as they make their own mistakes and then empathize with your past transgressions.

And in the AI era, this attitude has extended even toward code engineers aren’t writing themselves. Everyone agrees not to protest too loudly if someone commits something wrong because tomorrow it might be you who has done so. The days of “obviously bad” slop code are largely behind us and most recent LLMs tend to put out decent code, so it is very hard to catch problems in review phases, you’d have to be very eagle eyed and that’s not scalable to every employee. It is much easier to let the code run wild for a bit and just be ready to fight fires should they come up.