Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AI handles incidents, engineers lose touch with their systems

sylvainkalache.com

Post reply on HN
Print view View original

221–230 of 378 posts

Re: AI handles incidents, engineers lose touch with their systems

#221

Post by backlava12 »

DiggyJohnson wrote: post #42
mawadev wrote: post #25

Earlier quoted context omitted.

I can share a weird story: Usually, I take my time to understand each keyword of the code I'm looking at, especially if it is new to me, like terraform. I work in a team/with one architect, who only did the DevOps/Infra stuff for the past years and I had the expectation he knows what he is doing and talking about. At around 2 weeks, I noticed how his knowledge has severe gaps and how he takes things at face value or…

You captured this phenomenon very well in this comment. Appreciate you sharing it because it’s hard to describe exactly what makes this sort of behavior so bizarre.

[dead]

Re: AI handles incidents, engineers lose touch with their systems

#222

Post by solatic »

seer wrote: post #219
solatic wrote: post #112

Author has a good head on their shoulders, but few if any companies are going to spend time on incident simulations for their SREs. Why not? Because even pre-AI, very few companies spend time practicing restoring their backups, or disaster recovery, or picking infrequently-used runbooks to practice, or seeing whether they can easily rotate secrets without downtime, or trying to redploy the system onto another vendor'…

Hmm that’s strange, in my experience it is the other way around - Claude is super diligent with infra and will _insist_ on double checking and trying everything for real before committing. When I was doing this myself I would read the docs and just implement them - claud is going about doing real software archeology to figure if what is said is actually the truth or it’s stale/inaccurate/buggy. I’ve become 10 times m…

> Backups are _tested regularly_ now because LLMs make it cheap to do so.

sighs heavily in 90's sysadmin

Testing backups is not just a question of whether or not the restore command works. Go back and read the Tao of Backup: http://www.taobackup.com/history.html . The application itself (in its current version, with its current features) needs to work with the backed-up data, and the only way to verify this is to attempt to actually work with the data.

If you don't trust your agent to ship to production without manually reviewing the output (in some way), you have no business trusting your agent managing your backups. The agent writing some tests doesn't mean that the tests adequately handle all of your actual scenarios, let alone that your system will adequately handle data that is missing since the last backup.

Re: AI handles incidents, engineers lose touch with their systems

#223

Post by 2sk21 »

huurtehoog wrote: post #120
noir_lord wrote: post #108

Earlier quoted context omitted.

Pretty much, The one thing I use it for is as a sanity check, pretty much "Look at , point out issues you see, summarise them tersely" and it'll spot stuff a code review by a human might have spotted (in the mythical land where people actually do code reviews properly and don't just flag a spelling mistake to "show they looked at it"). Beyond that I don't trust it at all and I still write all my code the meat sack wa…

If anything, I think this hype cycle is fast exposing just how many people, teams, and companies just don't care about what is correct. They just wanna feel good about themselves and get paid. I for one welcome the fact that this whole thing has driven me back to books and deeper into the fundamentals. I have never read so much on math, hardware, and history as in the past 3 years or so.

Much the same here - I have gone back to basics and am studying a lot more than I ever did.

Re: AI handles incidents, engineers lose touch with their systems

#224

Post by acedTrex »

VCFundedGenYer wrote: post #207
konschubert wrote: post #61

Earlier quoted context omitted.

It's the uncanny valley of AI. It's still not quite good enough yet that you can trust it blindly on a big codebase, so you still have to read and understand everything - which is often harder than just writing it up yourself. EDIT: don’t get me wrong. I still think AI is incredibly useful for a lot of tasks! But when implementing an architecturally hairy thing, I find it less stressful and equally quick to jump down…

Nah, It's not even good for small code changes. Try using it with Ansible. It spits back complete buffoonery.

I was investigating an ansible playbook yesterday that had a 45 line comment to explain a single apt install command, completely and utterly useless. I am updating my neovim to just collapse all comments, the noise is unbearable.

Re: AI handles incidents, engineers lose touch with their systems

#225

Post by sidhsjsjhz »

nekoashide wrote: post #157
JauntyHatAngle wrote: post #123

Earlier quoted context omitted.

When I hear this I do wonder to myself how they're using AI. For me, as long as I'm properly RPI looping it and not blindly pressing "yes" then it will nearly always reach the solution, usually a fair bit quicker, because it effectively becomes an ideation machine that can keep more thoughts and knowledge in it than my brain can. However, I'm using it through giving it the context, it has access to the repos, access…

In my experience at a large bank with unlimited AI, my spend is in the top 5% and I'm leveraging AI just like you. I was in a meeting with someone who had a bug in an application that I don't own myself on Friday so I told claude, "I need you to find this bug the user is experiencing, find out if it's user error or a bug, let the user know and tell the developer what needs to be fixed if needed" 15 minutes later the…

> "I need you to find this bug the user is experiencing, find out if it's user error or a bug, let the user know and tell the developer what needs to be fixed if needed" 15 minutes later the developer ask me if I want it fixed today or Tuesday.

Did you find out if it actually fixed the issue? I was on the other end of this last week. Reported an issue to another team, they asked Claude to diagnose, and when they gave me the response back I pushed back cause it didn’t make sense given the behavior we were seeing. Turns out Claude had hallucinated reading a log (it said it did but it didn’t). Sent it down a hole rabbit hole from there.

I’ve had the one shots you describe and they’re great, but they’re the happy path and require almost 0 skill to find. I’ve yet to see a case where developing an expert level knowledge of your domain isn’t the best way to get good at LLMs. Knowledge of how to interface with these tools is helpful but changes (and depreciates) rapidly as the models get smarter and/or other people commoditize it. Deep subject knowledge is still the best way to get a lot out of these tools. This applies to development and other areas I have a deep knowledge of.

“LLM training” is kinda snake oil. Learn your craft deeply and you’ll be able to catch up on LLM training in a few days, but you’ll be light years ahead of a person who’s only expertise is different ways to prompt.

Re: AI handles incidents, engineers lose touch with their systems

#226

Post by oblio »

JauntyHatAngle wrote: post #199
oblio wrote: post #171

Earlier quoted context omitted.

> RPI loop What's that?

Research Plan Implement. At the risk of you already knowing this or similar - in essence for a slice of a piece of work, you get the LLM to research the information required into (usually) a research file, which you go back and forth with in-line or via conversation until you're happy. You then build the plan for the piece of work, again going back and forth with the LLM until you agree with the plan, with the key he…

Ah, that makes sense. I'm doing the same thing but I knew it as Spec Driven Development (SDD, I guess).

Re: AI handles incidents, engineers lose touch with their systems

#227

Post by w0m »

LeafItAlone wrote: post #204
Richbeach wrote: post #150

Earlier quoted context omitted.

> and it's quite enjoyable Used to be somewhat enjoyable. Nothing pleasant about digging around codebase that was heavily affected by the last 12-18 months of AI-ing.

Depends on the prior state. For one codebase I work in, LLM written code is noticeably and measurably (we have literally measured it for bugs, performance, etc.) better than what existed before. So the LLM code is a welcome change.

'ai slop' is hard for me to complain about when it's better than 95% of code prior. The issue is more people trusting it too much to go too far, and you end up with lots of extraneous bullcrap you don't need. Doing too much and too big of a change is still a problem in the agentic world.

Re: AI handles incidents, engineers lose touch with their systems

#229

Post by dayvid »

LeafItAlone wrote: post #204
Richbeach wrote: post #150

Earlier quoted context omitted.

> and it's quite enjoyable Used to be somewhat enjoyable. Nothing pleasant about digging around codebase that was heavily affected by the last 12-18 months of AI-ing.

Depends on the prior state. For one codebase I work in, LLM written code is noticeably and measurably (we have literally measured it for bugs, performance, etc.) better than what existed before. So the LLM code is a welcome change.

It also documents itself extensively

Re: AI handles incidents, engineers lose touch with their systems

#230

Post by 2sk21 »

spicyusername wrote: post #195

Its true we will lose the skills, but so far the LLMs are more than picking up the slack when wielded competently. They _regularly_ go above and beyond when troubleshooting and frequently in 1/2 - 1/1000th of the time. I remember spending entire days troubleshooting in the before times. Now it's like 30 minutes, tops, on literally any issue. This is what progress looks like. We used to do a thing and now we don't.

Who is going to teach programmers to wield AI competently?
Post reply on HN