Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Building better AI tools

hazelweakly.me

Post reply on HN
Print view View original

181–189 of 189 posts

Re: Building better AI tools

#181

Post by ofjcihen »

tptacek wrote: post #180
ofjcihen wrote: post #176

Earlier quoted context omitted.

In this case it’s relevant to the discussion as the user was questioning why you were making the points you were. It’s not an accusation of shilling, it’s context where context was requested. As a test imagine if you changed the context to something good such as “AI achieves the unthinkable” and the responding user asked why someone was so optimistic about the achievement. It’s relevant context to the conversation, n…

It's false context meant to impeach my arguments. Not a close call.

I promise that’s not the case but we can let it go for now.

Re: Building better AI tools

#182

Post by yunohn »

kookamamie wrote: post #167
cmiles74 wrote: post #71

Earlier quoted context omitted.

There's no AI tool today that will resolve incidents to anyone's satisfaction. People need to be in the loop not only to take responsibility but to make sure the right actions are performed.

Exactly. There seems to be this fantasy in which you can somehow string different kinds of agents together, one designing and one reviewing, and that finally producing something superior as output - I just don't buy that. Sounds like heuristics added on top of statistics, which is trying to remedy some root problem with another hack.

Hmm, but this provably works right now though? All LLMs perform better with roleplay direction and focused scope. Using coding agents with plan then execute makes noticeable quality improvements.

Re: Building better AI tools

#183

Post by shrumm »

tptacek wrote: post #54

This is a confusing piece. A lot of it would make sense if Weakly was talking about a coding agent (a particular flavor of agent that worked more like how antirez just said he prefers coding with AI in 2025 --- more manual, more advisory, less do-ing). But she's not: she's talking about agents that assist in investigating and resolving operations incidents. The fulcrum of Weakly's argument is that agents should stay…

> But what exactly is the value in having humans grovel through logs to isolate anomalies and create hypotheses for incidents?

Agreed! I think about this using Weakly's own reference to "standing on the shoulders of giants."

To me, building abstractions to handle tedious work is how we do that. We moved from assembly to compilers, and from manual memory management to garbage collectors. That wasn't "deskilling" - it just freed us up to solve more interesting problems at a higher level.

Manually crawling through logs feels like the next thing we should happily give up. It's painful, and I don't know many engineers who enjoy it.

Disclaimer: I'm very biased - working on an agent for this exact use case.

Re: Building better AI tools

#184

Post by tptacek »

otterley wrote: post #178
tptacek wrote: post #54

This is a confusing piece. A lot of it would make sense if Weakly was talking about a coding agent (a particular flavor of agent that worked more like how antirez just said he prefers coding with AI in 2025 --- more manual, more advisory, less do-ing). But she's not: she's talking about agents that assist in investigating and resolving operations incidents. The fulcrum of Weakly's argument is that agents should stay…

> There are actions AI tools shouldn't perform autonomously (I certainly wouldn't let one run a Terraform apply), but there are plenty of actions where it doesn't make sense to stop them. I'm curious as to where you would draw the line. Assuming you've adhered to DevOps best practices, most--if not all--changes would require some sort of code commit and promotion through successive environments to reach production. T…

During incident resolution, most of the actions an operator takes are diagnostic commands, not changes.

Re: Building better AI tools

#185

Post by kookamamie »

yunohn wrote: post #182
kookamamie wrote: post #167

Earlier quoted context omitted.

Exactly. There seems to be this fantasy in which you can somehow string different kinds of agents together, one designing and one reviewing, and that finally producing something superior as output - I just don't buy that. Sounds like heuristics added on top of statistics, which is trying to remedy some root problem with another hack.

Hmm, but this provably works right now though? All LLMs perform better with roleplay direction and focused scope. Using coding agents with plan then execute makes noticeable quality improvements.

Why isn't this the de facto then? Anyone packaging such commercial solutions?

Re: Building better AI tools

#186

Post by yunohn »

kookamamie wrote: post #185
yunohn wrote: post #182

Earlier quoted context omitted.

Hmm, but this provably works right now though? All LLMs perform better with roleplay direction and focused scope. Using coding agents with plan then execute makes noticeable quality improvements.

Why isn't this the de facto then? Anyone packaging such commercial solutions?

Most agents solutions have modes or roles already. There’s no standard, but this is already being used IRL. Heck, even system prompts are role play too.

Re: Building better AI tools

#187

Post by otterley »

tptacek wrote: post #184
otterley wrote: post #178

Earlier quoted context omitted.

> There are actions AI tools shouldn't perform autonomously (I certainly wouldn't let one run a Terraform apply), but there are plenty of actions where it doesn't make sense to stop them. I'm curious as to where you would draw the line. Assuming you've adhered to DevOps best practices, most--if not all--changes would require some sort of code commit and promotion through successive environments to reach production. T…

During incident resolution, most of the actions an operator takes are diagnostic commands, not changes.

The number one cause of incidents is change, and the number one response to them is to initiate a rollback. Maybe you’re right about investigation, which requires no changes, but resolution requires action, which does.

In any event, you said:

> What Weakly seems to be doing is laying out a bright line between advising engineers and actually performing actions --- any kind of action, other than suggestions (and only those suggestions the human driver would want, and wouldn't prefer to learn and upskill on their own). That's not the right line.

So what’s your quibble exactly? Those suggestions would come from autonomous analyses, would they not? What is the right line, in your view?

Re: Building better AI tools

#188

Post by tptacek »

otterley wrote: post #187
tptacek wrote: post #184

Earlier quoted context omitted.

During incident resolution, most of the actions an operator takes are diagnostic commands, not changes.

The number one cause of incidents is change, and the number one response to them is to initiate a rollback. Maybe you’re right about investigation, which requires no changes, but resolution requires action, which does. In any event, you said: > What Weakly seems to be doing is laying out a bright line between advising engineers and actually performing actions --- any kind of action, other than suggestions (and only t…

I would not in 2025 during an incident response have an agent do speculative changes, or really any changes at all.

I would have an agent perform diagnostic steps: dumping devicemapper tables, iproute2 configurations, nftables rules, BGP advertisements, Consul data, and, especially, logs and oTel telemetry.

Weakly's article is in large part about not allowing agents to do the things in the second category there.

Re: Building better AI tools

#189

Post by cwnyth »

bluefirebrand wrote: post #87
cwnyth wrote: post #76

Earlier quoted context omitted.

My LLM-generated code has so many bugs in it, that I end up knowing it better since I have to spend more time debugging/figuring out small errors. This might even be better: you learn something more thoroughly when you not only practice the right answers, but know how to fix the wrong answers.

That is absurd If you write it by hand you don't need to "learn it thoroughly", you wrote it There is no way you understand code between by reading it than by creating it. Creating it is how you prove you understand it!

Or, you can copy and paste code from examples, StackExchange, open source code, etc. Or you can read about it once, use it, and forget why it worked.

Besides all that, though, it's really the fact that LLMs bring up interesting ways to tackle problems that I hadn't thought of before, or uncover neat libraries/packages (when I program in R) that I just am not aware of.

Post reply on HN