Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The short leash AI coding method for beating Fable

blog.okturtles.org

Post reply on HN
Print view View original

141–150 of 268 posts

Re: The short leash AI coding method for beating Fable

#141

Post by taberiand »

codazoda wrote: post #62
1123581321 wrote: post #40

Earlier quoted context omitted.

I did it by making a huge database of allowlisted bash and having hooks check each one against the list. It makes a recursively parsed tree so it can handle gnarly blocks of bash. And then it outputs to the agent what failed and tells it to break it up next time. Then, in agent instructions, I impress on it strongly to use composable bash tools rather than trying to write python/ruby/perl scripts. It was a bit of wor…

This sounds like an interesting path. Wish I had time (instead of reading endless prompts and getting fatigued).

You could probably get Claude to build the plugin for you

Re: The short leash AI coding method for beating Fable

#142

Post by ergl »

ed_mercer wrote: post #34

I feel like OP is still in the year 2025. > The AI will have gone off the rails multiple times and you will only notice it later when you actually try to use the software. Except that said AI can now themselves use your software and find and fix bugs themselves, not to mention drive new features. >Your agent might go “off the rails” and start doing something you don’t want it to do This happens but far less often tha…

You're right, understanding things is so 2025.

Re: The short leash AI coding method for beating Fable

#143

Post by IsTom »

ACCount37 wrote: post #39
moezd wrote: post #18

LLMs are still next token predictors, just because you can give it more vague instructions and it still finds the right steps to follow, it doesn't mean it's intelligent. It means you're speaking the same language as the harness they trained your model on. And that has a limit. If you are stuck at PoC level or simple apps, you have no idea how limited the current models still are. There you really need to break tasks…

"Next token prediction" is an interface, not an algorithm. A process that "predicts next tokens" can be arbitrarily complex or simple, and arbitrarily capable or incapable of performing a given task. Saying that an LLM can or can't do something because it's a "token predictor" is a category error. The interface isn't a hard limit.

I'm not sure if it's has any real bearing on real-world performance, but technically next token prediction makes it an online algorithm and they can be provably worse than (good) offline algorithms.

Re: The short leash AI coding method for beating Fable

#145

Post by te_chris »

sothatsit wrote: post #126
RealityVoid wrote: post #110

Earlier quoted context omitted.

I am a bit confused which part you disagree with specifically. Reading AI responses and reviewing code seems to be what you propose as well. Your example with MLIP is something that would not be prevented by this approach, during the planing phase, it would surface. I guess the devil is in the details and the way you prompt it for starting the task matters. But IMO you absolutely need to check the output, need to eng…

I disagree with keeping an eye on the model as it is working, approving every command, and denying and stopping the model when you think it has gone wrong. It is not that it is actively harmful to do this, but rather that it is a waste of time and you can avoid the need for it through better design discussions and review. Micro-managing and keeping the AI on a "short leash" also lends itself better to telling models…

I agree. Better to let it rip in a sandbox then spend your time correcting the finished product.

Waste of time being in the middle.

Re: The short leash AI coding method for beating Fable

#147

Post by Zecc »

matheusmoreira wrote: post #47
ofjcihen wrote: post #31

Earlier quoted context omitted.

I love this argument. Not because it’s true but because it betrays the posters doubt in their own sentience.

It's impossible for someone to doubt their own sentience. The literal act of doubting is enough to dissipate all doubt. Solipsism is essentially the one certainty that every mind out there has. Doubting the sentience of machines and even other humans is perfectly fine though. Only empathy allows people to make the leap and assume other humans have souls.

> Solipsism is essentially the one certainty that every mind out there has.

Not I. I'm just a Boltzmann brain.

Re: The short leash AI coding method for beating Fable

#148

Post by ramon156 »

One problem I have with "how to do X with AI" is that every situation is different. For example, I'm bumping Symfony projects from 3.1 to 8.1. There's a clear path here

- Follow the written up migration guides PER major version

- test all routes, authorised, etc. You can even hand-curate these tests. some might return 200, some might return 302

- Maybe optionally start with writing a safety net so you do not need to do these test manually, have e.g. a PHPStan baseline, etc.

You're done when the routes are e2e functionally working as intended. You could even use snapshot testing here.

I do not need to look at the AI here. I can review the code at the end, but I do not need to manually approve stuff here, hence safety features are off.

Re: The short leash AI coding method for beating Fable

#149

Post by iainmerrick »

fny wrote: post #46

AI is a junior to mid-level engineer. If you treat it as such, you get the best of both vibe coding and rigorous engineering without all this paranoia. Since the very beginning I've ran Claude from an isolated VM on yolo mode. This is just like giving an engineer their own laptop. Claude works on a feature up to a PR worthy point. I review the diff, just like I would with another engineer, and massage it to get it in…

Yep, this is my experience too. I think of it more as a very, very smart and fast intern -- you can tell it’s going places, and in many ways is already way better than you, but it still needs an experienced hand to steer it.

My rule of thumb is, any special processes you put in place for AIs are either sensible for humans as well, or they’re not worthwhile. Good CLIs, auto-summarization of long command outputs, Markdown docs and workflows -- those are all useful for people too!

To guard against mistakes and abuse, you use sandboxing and scoped permissions, not micromanagement.

One thing I’d like to figure out is a good pair-programming workflow for AI agents. You can tell a high-level model to go and do something, and that works; you can use a low-level model as an IDE assistant, and that works; but they’re separate workflows. What would be really useful is a way to kind of hand the keyboard back and forth with the high-end model and build something together. But safely, not in full-on YOLO mode on my own machine. This is one specific area where humans and LLMs differ -- it’s so much faster than me that I can’t just grab the keyboard back from it if it goes off the rails.

Re: The short leash AI coding method for beating Fable

#150

Post by esailija »

infinite_spin wrote: post #35
moezd wrote: post #18

LLMs are still next token predictors, just because you can give it more vague instructions and it still finds the right steps to follow, it doesn't mean it's intelligent. It means you're speaking the same language as the harness they trained your model on. And that has a limit. If you are stuck at PoC level or simple apps, you have no idea how limited the current models still are. There you really need to break tasks…

> it doesn't mean it's intelligent I'm not sure how you're defining "intelligent", but I'd like to know how it is able to exclude a language model, while still including humans, without simply defining it with an axiom that predefines LLMs as lacking intelligence.

Intelligence is the complete opposite of an LLM. Usually the more you needed to memorize to do something the less intelligent you were considered.

It was also not considered to be a different route to the same thing, but more like fraud.

Also conceptually I could just write the weights on paper and do the billion multiplications on paper without any computer, does that mean I am the paper or the numbers or what??

Post reply on HN