Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Building better AI tools

hazelweakly.me

Post reply on HN
Print view View original

161–170 of 189 posts

Re: Building better AI tools

#161

Post by bubblyworld »

tptacek wrote: post #54

This is a confusing piece. A lot of it would make sense if Weakly was talking about a coding agent (a particular flavor of agent that worked more like how antirez just said he prefers coding with AI in 2025 --- more manual, more advisory, less do-ing). But she's not: she's talking about agents that assist in investigating and resolving operations incidents. The fulcrum of Weakly's argument is that agents should stay…

I know you've got a subthread about this exact idea, but I do think there is some value in manually performing the debugging process if (and perhaps only if) your goal is to improve your overall programming ability.

I guess the chess analogy would be that it makes a lot of sense to analyse positions yourself, even though Leela and Stockfish can do a far more thorough job in much less time. Of course, if you just need to know the best move right now, you would use the AI, and professionals do that all the time.

But as a decently strong chess player I cannot imagine improving without doing this kind of manual practice (at least beyond a basic level of skill like knowing how pieces move). Grandmasters routinely drill tactics exercises, for instance, even though they are "mundane" at that level of ability.

I guess the crux of it - do you think AI+person learns faster than just person for this kind of thing? And why? It's not obvious to me either way (and another question is whether the skill is even relevant any more... I think so, but I know people who don't).

Re: Building better AI tools

#162

Post by hiAndrewQuinn »

tptacek wrote: post #54

This is a confusing piece. A lot of it would make sense if Weakly was talking about a coding agent (a particular flavor of agent that worked more like how antirez just said he prefers coding with AI in 2025 --- more manual, more advisory, less do-ing). But she's not: she's talking about agents that assist in investigating and resolving operations incidents. The fulcrum of Weakly's argument is that agents should stay…

>The fulcrum of Weakly's argument is that agents should stay in their lane, offering helpful Clippy-like suggestions and letting humans drive. But what exactly is the value in having humans grovel through logs to isolate anomalies and create hypotheses for incidents?

See also: Tool AIs Want To Be Agent AIs.

https://gwern.net/tool-ai

Predicted almost a decade ago.

Re: Building better AI tools

#163

Post by rossant »

jgb1984 wrote: post #146
taylorallred wrote: post #14

One thing that has always worried me about AI coding is the loss of practice. To me, writing the code by hand (including the boilerplate and things I've done hundreds of times) is the equivalent of Mr. Miyagi's paint-the-fence. Each iteration gets it deeper into your brain and having these patterns as a part of you makes you much more effective at making higher-level design decisions.

What worries me more is the steep decline in code quality. The python and javascript output I've seen the supposed best LLM's generate is inefficient, overly verbose and needlessly commented at best, and simply full of bugs at worst. In the best case they're glaringly obvious bugs, in the worst case they're subtle ones that will wreak havoc for a long time before they're eventually discovered, but by then the grasp o…

I'm very cautious about using LLM-generated code in production, but for one-off throwaway scripts that generate output I can manually verify, LLMs are a huge time saver.

Re: Building better AI tools

#164

Post by machiaweliczny »

lubujackson wrote: post #19

Great insights. Specifically inverting the vibe coding flow to start with architecture and tests is 100% more effective and surfaceable into a real code base. This doesn't even require any special tooling besides changing your workflow habits (though tooling or standardized prompts would help).

Yeah, I started creating my own architect tool as this is what missing currently. Given good architecture you can really hand down implementation to AI these days. One problem I see that these tools aren't good at reading logs of long running processes (like docker-compose)

But you need to: * Research problems * Describe features * Define API contracts * Define basic implementation plan * Setup credentials * Provide testing strategy and setup efficient testing setup/teardown * Define libraries docs and references and find legit documentation for AI * Also AI does a lot mistakes with imports etc. and long running processes

Re: Building better AI tools

#165

Post by kasey_junk »

bubblyworld wrote: post #161
tptacek wrote: post #54

This is a confusing piece. A lot of it would make sense if Weakly was talking about a coding agent (a particular flavor of agent that worked more like how antirez just said he prefers coding with AI in 2025 --- more manual, more advisory, less do-ing). But she's not: she's talking about agents that assist in investigating and resolving operations incidents. The fulcrum of Weakly's argument is that agents should stay…

I know you've got a subthread about this exact idea, but I do think there is some value in manually performing the debugging process if (and perhaps only if) your goal is to improve your overall programming ability. I guess the chess analogy would be that it makes a lot of sense to analyse positions yourself, even though Leela and Stockfish can do a far more thorough job in much less time. Of course, if you just need…

But you can do that _after_ the incident. When things are not on fire.

You don’t run analysis of your chess game when the clock is ticking.

Re: Building better AI tools

#166

Post by bubblyworld »

kasey_junk wrote: post #165
bubblyworld wrote: post #161

Earlier quoted context omitted.

I know you've got a subthread about this exact idea, but I do think there is some value in manually performing the debugging process if (and perhaps only if) your goal is to improve your overall programming ability. I guess the chess analogy would be that it makes a lot of sense to analyse positions yourself, even though Leela and Stockfish can do a far more thorough job in much less time. Of course, if you just need…

But you can do that _after_ the incident. When things are not on fire. You don’t run analysis of your chess game when the clock is ticking.

Sure, if something is super critical then you should solve the problem as fast as possible. I'm not debating that. But there's probably a middle ground there somewhere for less critical issues. I suspect the process of generating and falsifying hypotheses quickly is the skill, and I don't know if you can effectively train that skill after an incident, when you've already seen the resolution.

Chess is maybe not a great analogy, because there are rarely objectively correct answers, only hard trade-offs. For that reason there's still a lot of value in reviewing a finished game.

Re: Building better AI tools

#167

Post by kookamamie »

cmiles74 wrote: post #71
tptacek wrote: post #54

This is a confusing piece. A lot of it would make sense if Weakly was talking about a coding agent (a particular flavor of agent that worked more like how antirez just said he prefers coding with AI in 2025 --- more manual, more advisory, less do-ing). But she's not: she's talking about agents that assist in investigating and resolving operations incidents. The fulcrum of Weakly's argument is that agents should stay…

There's no AI tool today that will resolve incidents to anyone's satisfaction. People need to be in the loop not only to take responsibility but to make sure the right actions are performed.

Exactly. There seems to be this fantasy in which you can somehow string different kinds of agents together, one designing and one reviewing, and that finally producing something superior as output - I just don't buy that.

Sounds like heuristics added on top of statistics, which is trying to remedy some root problem with another hack.

Re: Building better AI tools

#169

Post by rusticpenn »

kookamamie wrote: post #167
cmiles74 wrote: post #71

Earlier quoted context omitted.

There's no AI tool today that will resolve incidents to anyone's satisfaction. People need to be in the loop not only to take responsibility but to make sure the right actions are performed.

Exactly. There seems to be this fantasy in which you can somehow string different kinds of agents together, one designing and one reviewing, and that finally producing something superior as output - I just don't buy that. Sounds like heuristics added on top of statistics, which is trying to remedy some root problem with another hack.

The whole field of metaheuristic algorithms rests on a similar idea. a lot of stupid "agents" finding a good solution. by metaheuristics i mean genetic algorithms, PSO, ACO etc.

Re: Building better AI tools

#170

Post by jakebennet89 »

tptacek wrote: post #127
jpc0 wrote: post #123

Earlier quoted context omitted.

I’m going to but in here. I think the point the article is making is to observe the patterns humans (hopefully good ones) follow to resolve issues and build paths to make that quicker. So at first the AI does almost nothing, it observes that in general the human will search for specific logs. If it observes that behaviour enough it then, on its own or through a ticket, builds a Ui flow that enables that behaviour. So…

We're just disagreeing and hashing this out, but, no, I don't think that's accurate. AI tools don't watch what human operators in a specific infrastructure do and then try to replicate them. They do things autonomously based on their own voluminous training information, and those things include lots of steps that humans are unlikely to take, and that are useful. One intuitive way to think about this is that any human…

[dead]
Post reply on HN