Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The short leash AI coding method for beating Fable

blog.okturtles.org

Post reply on HN
Print view View original

251–260 of 268 posts

Re: The short leash AI coding method for beating Fable

#251

Post by lambdaone »

loose-cannon wrote: post #250
lambdaone wrote: post #247

Earlier quoted context omitted.

Even asking a human being why they did a certain thing is questionable. The research on choice blindness seems like a pretty definitive debunking of post-hoc rationalization: https://en.wikipedia.org/wiki/Introspection_illusion#Choice_...

I'm not sure what point you're trying to make. In science and engineering, being able to provide justification is a core skill. The comparison we should be making is against the human practitioners who are trained in their fields. There will always be a distribution of ability. Saying that there's evidence that people are capable of providing post-hoc rationalization doesn't say anything about the ability of experts…

Structured thinking and deliberation are indeed important, but you can also make LLMs do structured "thinking" if you work hard enough, and generate quite plausible reasoned arguments with valid real-world results, and you can get them to write down their working as they go. But as research has shown, it's not "true" thinking, just pattern matching at a higher level, and eventually runs out of steam.[0]

But you only have to drill down a couple more layers and you are back in the void again; do you have any proof that your own thinking, no matter how structured and accurate, is anything other than pattern-matching at a sufficiently much higher level at which you are incapable of seeing it as such?

I think we will be finding some very interesting things out soon using the combination of LLMs and theorem provers, as demonstrated by Terence Tao's recent work.[1]

A cheetah is not a motorbike is not an aircraft is not a rocket.

[0] https://arxiv.org/abs/2506.06941

[1] https://arxiv.org/abs/2603.12744

Re: The short leash AI coding method for beating Fable

#252

Post by weakfish »

cpursley wrote: post #233
weakfish wrote: post #221

Earlier quoted context omitted.

> turd bucket autist I’d be more willing to engage with your argument in good faith without inflammatory language like this. Try and meet people where they are and these conversations become easier.

[flagged]

I’d prefer kindness and good faith when talking to strangers, but maybe my expectations are too high.

Do you think you’ll change someone’s mind by being an asshole? Rarely works.

Re: The short leash AI coding method for beating Fable

#253

Post by eithed »

Mostly agree with the author. Would add, most importantly, dont trust anything LLM does or says. Today I asked Claude to uniform behaviour of 3 components. I asked to do it 5 times, because at the end of each go there was something still unaligned that Claude found a way to rationalize. Sure - when asked the way it would say "This is on me", or "I thought it was a concious choice". Not once did it surface a question on what to do, nor did it mention any of the issues. So yeah, short leash, look at its thinking, correct his shit. This is today, Sonnet 5. Probably tomorrow it will be better or worse - thats another thing. The way you talk to Claude today will give you different results tomorrow

Re: The short leash AI coding method for beating Fable

#254

Post by Toutouxc »

solenoid0937 wrote: post #111
sixtram wrote: post #106

Earlier quoted context omitted.

You say you can have increasingly nuanced discussions with stronger models. What I say is, when I asked Claude why he applied a certain change I didn't understand, and boy, it was a small change, he said he "reasoned from first principles" based on the code paths. But it didn't work, and when I asked, "Okay, describe the steps of your reasoning from first principles," it literally answered that it had just made it up…

Posts like this are meaningless without more context - the model you're using, the harness, the initial prompt and context. Fable is better than most staff engineers at my FAANG.

> Fable is better than most staff engineers at my FAANG.

That’s genuinely disturbing.

Re: The short leash AI coding method for beating Fable

#255

Post by Toutouxc »

seunosewa wrote: post #186
chewbacha wrote: post #94

I did this for two weeks on a side project and still ended up in a situation where I did not have a mental model of the codebase. There’s no way build that model without building it yourself. I’m more convinced then ever of this.

You can just ask the model to explain the code to you.

For me, the issue is that the model can explain what the code IS, not what it’s supposed to be or aspiring to be. The code doesn’t explain the original constraints (because it already fits within them) and it doesn’t show the tried and discarded approaches. It’s like rewriting a MIDI roll back into sheet music. Or better, it’s like the difference between being the author of an opera, and having read reviews of it.

Re: The short leash AI coding method for beating Fable

#256

Post by Toutouxc »

esperent wrote: post #174
chewbacha wrote: post #94

I did this for two weeks on a side project and still ended up in a situation where I did not have a mental model of the codebase. There’s no way build that model without building it yourself. I’m more convinced then ever of this.

If you were working as a manager on a large project, how would you build a model? Something where your position requires you to have an overview of the project but not necessarily to actually write or review much code.

My understanding is that this is still an unsolved problem. At some point you lose touch with the code and you start working with a mental model that’s subtly inaccurate. You just sort of hope that the people below have are competent and are building stuff that matches your mental model (that you’re communicating well!).

Re: The short leash AI coding method for beating Fable

#257

Post by chaboud »

This could also read as "how to be a horrible people manager for junior engineers".

Techniques that work for inexperienced engineers with high ability but limited judgment often work well with agentic coding systems.

- Give them clarity of purpose. Why are they doing what they're doing?

- Make explaining it back to you part of the job.

- Give them two-way doors. Make mistakes reversible.

- Put effort into thoughtful refactoring as an actual sub-task instead of just accepting piled on hacks.

- Make your operating rules crisp and make sure they store them in their memories.

- Be accountable for their work. It's not okay to crank out AI Slop and then say "Claude's fault".

We're all Software Development Managers now.

So, micromanage the LLMs if you want to, but you'll be missing out on chances to improve them for your purposes and, more importantly, to improve yourself as a manager.

Re: The short leash AI coding method for beating Fable

#259

Post by seunosewa »

Toutouxc wrote: post #255
seunosewa wrote: post #186

Earlier quoted context omitted.

You can just ask the model to explain the code to you.

For me, the issue is that the model can explain what the code IS, not what it’s supposed to be or aspiring to be. The code doesn’t explain the original constraints (because it already fits within them) and it doesn’t show the tried and discarded approaches. It’s like rewriting a MIDI roll back into sheet music. Or better, it’s like the difference between being the author of an opera, and having read reviews of it.

One of my favourite things to do is to ask the models, "what does this code do and why?" They are usually not far from the truth. From the perspective of a LLM that has been trained on all of the public code on GitHub, issues, and pull requests, your code and its intents are not that unique. They can see patterns in the code that point to the purpose. The constraints that shaped your code are the same constraints everyone else had.

Developer intents and aspirations aren't unique either. And in an LLM-written codebase, the chat history and prompts are available. The constraints, the rejected approaches, the goals. Users ask the models to write MD files that directly state their intents and aspirations for the ode.

Re: The short leash AI coding method for beating Fable

#260

Post by dizhn »

entropyneur wrote: post #137

This reminds me of the workflow I had a year ago. Miss Aider so much. Are there any good open source agents right now? Might be a good time to try one soon as Fable switches to token-based billing, which Code is designed to maximize.

Aider is still active. The new lighweight darling is pi. Complete batteries included solution, opencode. Turbocharge all of these and work on multiple harnesses at once, allowing every model to talk etc. Paseo. (Best mobile client too)
Post reply on HN