Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The short leash AI coding method for beating Fable

blog.okturtles.org

Post reply on HN
Print view View original

101–110 of 268 posts

Re: The short leash AI coding method for beating Fable

#101

Post by solumunus »

giancarlostoro wrote: post #56

Here I thought this was about Fable the video game, then I remembered Anthropics model got named Fable. It's going to be painful to google one of my favorite game series, just like googling "Rust server" does not give you Rust programming results, but Rust the video game results. I wish google would have fixed this problem long ago, it seems like something trivial for them to fix.

You want Google to be able to know which Fable you’re interested in when you type “fable”? Sir this seems unreasonable.

Re: The short leash AI coding method for beating Fable

#102

Post by echelon »

sothatsit wrote: post #23

This “short leash” seems like more of a crutch to me, and a sign of not giving the AI enough detail on the problem to begin with, or not reviewing and iterating on its output. Hand-holding great models like Fable through implementation is a waste of time, and a waste of Fable. You can have increasingly nuanced discussions with stronger models, and they write a lot better code than they used to. The process of discuss…

This is the method I use.

It makes sure that I understand everything being generated and that I maintain a firm working knowledge of the codebase at all times.

I can easily steer it too.

Re: The short leash AI coding method for beating Fable

#103

Post by hansvm »

bpodgursky wrote: post #96
hansvm wrote: post #93

Earlier quoted context omitted.

And who can't code its way out of a wet paper bag on hard problems. It's more productive for the day-to-day BS, which is convenient because it creates more day-to-day BS you need to handle, but that isn't the reason I hire a staff engineer.

i'm sorry but you're wrong and the only person you're hurting with your delusions is yourself. it doesn't change reality to pretend the world isn't changing under your feet. i'm not going to argue about this but for your own career etc i truly hope you evaluate your epistemics.

Sure, it's changing, and I use AI a ton. The second I ask it (where "it" is a smattering of all the SOTA models and harnesses) to do something as simple as design a server capable of doing when any concurrent data structures are involved and the single-server load is in the 100k QPS range, even with extremely thorough plans of how concurrency needs to be managed, it doesn't matter how little code is actually needed or how easy it would be for my juniors to bang out the problem, especially with a little AI boost, AI just can't keep up by itself yet. It can sometimes spit out something close, but only with major correctness issues.

I'm not trying to be argumentative; You posed an idea, and it looked wrong in an important way, so I added my observations. I'd love if you could share the model/harness/workflow you use that makes you so confident in this tooling, because I don't want to be left behind.

Re: The short leash AI coding method for beating Fable

#104

Post by fy20 »

CodingJeebus wrote: post #88
Foxhuls wrote: post #66

Earlier quoted context omitted.

I can't help but feel that this reads more as a reflection that you don't want to stop being a developer than it does that thing's aren't moving in the direction that the GP said it is.

Maybe, it seems like a bad idea for so many reasons though. Take away tactile code review, insert a layer of prompts and tooling between developers and the codebase, and you've created the conditions to let all kinds of nefarious things happen in a codebase. A disgruntled employee updates agent prompts instructing the code review bot to ignore data exfiltration vulnerabilities (because if we aren't reviewing code, we…

If you are just shipping code blindly without reviewing anything then that's your fault. My company heavily uses AI (I'd say 90% of code is written with AI assistance) but we never ship anything that hasn't been reviewed by a human.

This is how we use it for code reviews:

- a skill tells the agent to automatically run a subset of tests and linting before each commit

- another skill tells it to review the entire changeset before creating a PR, this review has more extensive rules that can't easily be put into code (e.g. linter rules) based on PR comments humans have written. It also sometimes catches things that were missed from the original prompt/task.

- when the PR is created we run a few AI tools to do automated code and security reviews. CI runs at the same time.

- the agent waits for these to complete, and verifies and fixes any issues if they are valid

- after all that it's passed back to the author to review

- once they are happy it's passed to a teammate to review

So we are not handing off reviews to AI, we are using it to do much more extensive reviews, and automatically fix stupid stuff the AI or human might have done. So by the time you are asked to review a PR, it should be pretty much ready to go, you can focus on what it's actually changing instead of looking for slop.

Re: The short leash AI coding method for beating Fable

#105

Post by andreygubarev »

ed_mercer wrote: post #34

I feel like OP is still in the year 2025. > The AI will have gone off the rails multiple times and you will only notice it later when you actually try to use the software. Except that said AI can now themselves use your software and find and fix bugs themselves, not to mention drive new features. >Your agent might go “off the rails” and start doing something you don’t want it to do This happens but far less often tha…

> This again feels outdated. I think we're mving towards humans no longer needing to understand a codebase, and letting AI drive it.

Seems so, but that doesn't mean it's a good or correct direction. As of today, none of the existing models can meaningfully handle mid-size tasks on five services with 10k+ LOC each, plus infra (I'm really not interested in greenfield projects done over the weekend that were never touched by actual users). It doesn't make them useless, but it significantly reduces the scope of trustworthy operations models can handle (unless you don't care about outcomes).

The moment your spec, plan, and results of related codebase exploration go beyond 100k tokens (roughly 50% of available context), quality degradation becomes real. Threads/subagents can help, and you can argue that code reviews mitigate some issues, but that's transitioning from reliable automation to gambling without human oversight. Say you want to mitigate the risks of failures (correctly listed by others) - how would you do that if you don't understand your codebase? In my practice, the answer is: you start to learn what your agents created, discover shit they created, and steer them toward better, desired outcomes.

Re: The short leash AI coding method for beating Fable

#106

Post by sixtram »

sothatsit wrote: post #23

This “short leash” seems like more of a crutch to me, and a sign of not giving the AI enough detail on the problem to begin with, or not reviewing and iterating on its output. Hand-holding great models like Fable through implementation is a waste of time, and a waste of Fable. You can have increasingly nuanced discussions with stronger models, and they write a lot better code than they used to. The process of discuss…

You say you can have increasingly nuanced discussions with stronger models.

What I say is, when I asked Claude why he applied a certain change I didn't understand, and boy, it was a small change, he said he "reasoned from first principles" based on the code paths. But it didn't work, and when I asked, "Okay, describe the steps of your reasoning from first principles," it literally answered that it had just made it up.

So, nuanced discussions with models, I don't buy it.

Re: The short leash AI coding method for beating Fable

#107

Post by zquzra »

sothatsit wrote: post #23

This “short leash” seems like more of a crutch to me, and a sign of not giving the AI enough detail on the problem to begin with, or not reviewing and iterating on its output. Hand-holding great models like Fable through implementation is a waste of time, and a waste of Fable. You can have increasingly nuanced discussions with stronger models, and they write a lot better code than they used to. The process of discuss…

Do you have a background in CS or optimization? MILP is a pretty standard concept in algorithms/optimization. So this example doesn't really convince me that the AI reached some unusually superior conclusion. It sounds more like it suggested a well-known technique that you personally hadn't encountered. Useful, yes, but that seems more about background knowledge gaps than about the merits of letting the tool run unconstrained.

Re: The short leash AI coding method for beating Fable

#109

Post by programmarchy »

Good luck with that. I used to be an OCD freak about code before LLMs, but AI coding has largely freed me of that limitation. I've become very comfortable giving AI a long leash, but only after being meticulous about curating the context.

These days I spend most of the day in discussions and planning, producing documentation, agonizing over architectural decisions, edge cases, and naming conventions. Once that's all settled I'll hand off implementation work to run overnight. In the morning, I'll review and fix, but I'm usually pleasantly surprised with the results.

One pitfall is long leash without a curated context, which is more like "slot machine" coding. Usually not effective, and may have addictive effects since it does occasionally work.

To spice things up lately, I've been encouraging the model to produce its own "capstone" -- a feature it decides to build on its own, however it wishes, with the tools at its disposal. So far it's been conservative, creating useful tools for development rather than customer facing features, but I'm curious to dial up the temperature to see what it might come up with.

Re: The short leash AI coding method for beating Fable

#110

Post by RealityVoid »

sothatsit wrote: post #23

This “short leash” seems like more of a crutch to me, and a sign of not giving the AI enough detail on the problem to begin with, or not reviewing and iterating on its output. Hand-holding great models like Fable through implementation is a waste of time, and a waste of Fable. You can have increasingly nuanced discussions with stronger models, and they write a lot better code than they used to. The process of discuss…

I am a bit confused which part you disagree with specifically. Reading AI responses and reviewing code seems to be what you propose as well.

Your example with MLIP is something that would not be prevented by this approach, during the planing phase, it would surface.

I guess the devil is in the details and the way you prompt it for starting the task matters.

But IMO you absolutely need to check the output, need to engage with what the model is doing, need to probe why something is built the way the model tries to build it.

Post reply on HN