Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The short leash AI coding method for beating Fable

blog.okturtles.org

Post reply on HN
Print view View original

111–120 of 268 posts

Re: The short leash AI coding method for beating Fable

#111

Post by solenoid0937 »

sixtram wrote: post #106
sothatsit wrote: post #23

This “short leash” seems like more of a crutch to me, and a sign of not giving the AI enough detail on the problem to begin with, or not reviewing and iterating on its output. Hand-holding great models like Fable through implementation is a waste of time, and a waste of Fable. You can have increasingly nuanced discussions with stronger models, and they write a lot better code than they used to. The process of discuss…

You say you can have increasingly nuanced discussions with stronger models. What I say is, when I asked Claude why he applied a certain change I didn't understand, and boy, it was a small change, he said he "reasoned from first principles" based on the code paths. But it didn't work, and when I asked, "Okay, describe the steps of your reasoning from first principles," it literally answered that it had just made it up…

Posts like this are meaningless without more context - the model you're using, the harness, the initial prompt and context.

Fable is better than most staff engineers at my FAANG.

Re: The short leash AI coding method for beating Fable

#112

Post by moezd »

infinite_spin wrote: post #35
moezd wrote: post #18

LLMs are still next token predictors, just because you can give it more vague instructions and it still finds the right steps to follow, it doesn't mean it's intelligent. It means you're speaking the same language as the harness they trained your model on. And that has a limit. If you are stuck at PoC level or simple apps, you have no idea how limited the current models still are. There you really need to break tasks…

> it doesn't mean it's intelligent I'm not sure how you're defining "intelligent", but I'd like to know how it is able to exclude a language model, while still including humans, without simply defining it with an axiom that predefines LLMs as lacking intelligence.

Intelligent humans are capable of following diverse and intricate analogies and draw lessons from seemingly unrelated events. Try asking an LLM to summarize an article and use an imprecise way to state your view. Ask it to push back. You will be drawn into so many pedantic arguments that burn through your tokens within a few messages, you'd wonder if there's someone deliberately taking over the keyboard on their side and spending your token limit. This would never happen with an intelligent human being unless they have nothing better to do and want to troll. This is a speech pattern that LLMs are trained on, it's not a show of intelligence. This also applies to LLMs claiming consciousness: The internet is full of people writing about sentience, talking to "superior aliens" in blog posts, forum threads etc. It's the speech pattern that's copied, not actual thoughts and feelings because LLMs perceive, suffer, have aims or dreams...

Re: The short leash AI coding method for beating Fable

#113

Post by solenoid0937 »

lowbloodsugar wrote: post #51
gambiting wrote: post #13

Earlier quoted context omitted.

>>You never use “YOLO” mode (aka “dangerously skip permissions”) Do you mean this? I'm curious how are people using Claude in any way other than bypass-permissions. I've tried for so long to maintain a curated list of things Claude can use, but inevitably I would always come back only to find it stuck because it decided to pipe an output of one tool into another and that's not explicitly allowed so it stopped even th…

Build your own MCP of allowed tools. Cargo. Ripgrep. File read and write, including directory listing and find. some git commands. Then block everything else.

Terrible advice. Turn on the sandbox, limit network connections, and let 'er rip.

Re: The short leash AI coding method for beating Fable

#114

Post by solenoid0937 »

andreygubarev wrote: post #105
ed_mercer wrote: post #34

I feel like OP is still in the year 2025. > The AI will have gone off the rails multiple times and you will only notice it later when you actually try to use the software. Except that said AI can now themselves use your software and find and fix bugs themselves, not to mention drive new features. >Your agent might go “off the rails” and start doing something you don’t want it to do This happens but far less often tha…

> This again feels outdated. I think we're mving towards humans no longer needing to understand a codebase, and letting AI drive it. Seems so, but that doesn't mean it's a good or correct direction. As of today, none of the existing models can meaningfully handle mid-size tasks on five services with 10k+ LOC each, plus infra (I'm really not interested in greenfield projects done over the weekend that were never touch…

> As of today, none of the existing models can meaningfully handle mid-size tasks on five services with 10k+ LOC each

My FAANG's codebase is a few orders of magnitude larger and agents do an excellent job of handling mid sized tasks completely autonomously.

Re: The short leash AI coding method for beating Fable

#115

Post by RealityVoid »

chewbacha wrote: post #94

I did this for two weeks on a side project and still ended up in a situation where I did not have a mental model of the codebase. There’s no way build that model without building it yourself. I’m more convinced then ever of this.

I'm not so sure. I think you can, you just need to intentionally drill into what you don't understand and it's exhausting. What I do agree with though is that I can't seem to build the ability to build it myself the same way as I would if I wrote it.

For example, I know my mental model works because I know what change I should do in order to get an effect and when I do the change, I get what I expect. But if I were to build myself something similar, I could not build it because the approach is somewhat out of my reach, I know it sounds weird, but it's hard to explain.

Re: The short leash AI coding method for beating Fable

#116

Post by semilin »

sixtram wrote: post #106
sothatsit wrote: post #23

This “short leash” seems like more of a crutch to me, and a sign of not giving the AI enough detail on the problem to begin with, or not reviewing and iterating on its output. Hand-holding great models like Fable through implementation is a waste of time, and a waste of Fable. You can have increasingly nuanced discussions with stronger models, and they write a lot better code than they used to. The process of discuss…

You say you can have increasingly nuanced discussions with stronger models. What I say is, when I asked Claude why he applied a certain change I didn't understand, and boy, it was a small change, he said he "reasoned from first principles" based on the code paths. But it didn't work, and when I asked, "Okay, describe the steps of your reasoning from first principles," it literally answered that it had just made it up…

"Nuanced discussion" doesn't necessarily mean the sort one would have with a human. Statistical apologies are never going to be meaningful. One could edit nonsense into the context window and the model would attempt to rationalize it. The models are smart but you need to use them in a way that makes sense for what they are.

Re: The short leash AI coding method for beating Fable

#117

Post by visarga »

densekernel wrote: post #27
sothatsit wrote: post #23

This “short leash” seems like more of a crutch to me, and a sign of not giving the AI enough detail on the problem to begin with, or not reviewing and iterating on its output. Hand-holding great models like Fable through implementation is a waste of time, and a waste of Fable. You can have increasingly nuanced discussions with stronger models, and they write a lot better code than they used to. The process of discuss…

I tend to agree, If you have invested significantly in the planning phase and there is momentum in the architecture and conventions that already exist in the project, the implementation phase might not need as much oversight as is suggested here. > You can discover that your initial idea was dumb and a better one exists The planning and architecture phase is usually where I make these types of discovery at a high lev…

I think the obvious solution here is to beef up the test side of the app, much more than when writing code by hand. Tests represent project knowledge in executable format. The LLM does not need to be careful to remember every detail of the tests. You don't need to vet every small interaction, it automates review work as well.

Even better if the project was built from the start to be easier to test and observe. But my golden rule remains - no code without tests, expand test suite all the time.

Re: The short leash AI coding method for beating Fable

#118

Post by moezd »

semiquaver wrote: post #29
moezd wrote: post #18

LLMs are still next token predictors, just because you can give it more vague instructions and it still finds the right steps to follow, it doesn't mean it's intelligent. It means you're speaking the same language as the harness they trained your model on. And that has a limit. If you are stuck at PoC level or simple apps, you have no idea how limited the current models still are. There you really need to break tasks…

Yeah, and you’re just a next-word-sayer.

Chinese whispers, simulacra... I don't have the energy to argue after being name called, but you get the point. Yes LLMs are useful in building automatic telling machines, but ask it to do anything more substantial and all you are doing is burning tokens at the altar of Anthropic and hope. That just doesn't fly in regulated industries.

Re: The short leash AI coding method for beating Fable

#119

Post by anon373839 »

ed_mercer wrote: post #34

I feel like OP is still in the year 2025. > The AI will have gone off the rails multiple times and you will only notice it later when you actually try to use the software. Except that said AI can now themselves use your software and find and fix bugs themselves, not to mention drive new features. >Your agent might go “off the rails” and start doing something you don’t want it to do This happens but far less often tha…

> I think we're mving towards humans no longer needing to understand a codebase, and letting AI drive it.

The AI companies are incentivized to push this kind of reckless slopmaxxing - the end result is that your business is totally dependent on them and your product's value entirely sourced from them. And a lot of people are buying it, but I think it's a silly fad.

Post reply on HN