I have found a different model should be used to do the review - like if Claude did the code, Codex should review. Models reviewing their own code is a recipe for disaster.
The short leash AI coding method for beating Fable
131–140 of 268 posts
Re: The short leash AI coding method for beating Fable
#132I'm curious whether Opus4.8 or similar can attain Mythos level through good system prompting and steering? You would expect this to work if it's true that the strength of Mythos is its unwillingness to quit before it gets a desired outcome
As a Mythos user (I’m part of Project Glasswing), I would say that abliterated models [1][2] produce similar, if not identical, results. While good prompting and steering won’t give Claude Opus 4.8 the same capabilities as Mythos (preview 1), using abliterated models (if you have the computational power to run the larger ones) will get you close to the same goals as people who have access to Mythos (preview 1) [3]. […
Are you saying similar-level in terms of security scans or is this something you think is true across all capabilities?
Re: The short leash AI coding method for beating Fable
#133I feel like OP is still in the year 2025. > The AI will have gone off the rails multiple times and you will only notice it later when you actually try to use the software. Except that said AI can now themselves use your software and find and fix bugs themselves, not to mention drive new features. >Your agent might go “off the rails” and start doing something you don’t want it to do This happens but far less often tha…
> Except that said AI can now themselves use your software and find and fix bugs themselves, not to mention drive new features.
Anyone with sufficiently good taste in how to program effectively and architect will disagree with you on this. The short leash method is how you ensure good results when you're functioning outside of the training data. If you're even a modestly above average programmer this is afaik the only way to ensure fast, quality development with LLMs.
> This again feels outdated. I think we're mving towards humans no longer needing to understand a codebase, and letting AI drive it.
I think you are perhaps unaware of a world of programming where AI is still woefully inept. I have observed very consistently in all languages with manual memory management frequent issues with handling it. Trust me, it's not as simple as sticking it in a loop with Valgrind.
Re: The short leash AI coding method for beating Fable
#134To me a lot of the anti-short leash sentiment is reflective of the low accountability SWE have always had for their output. Software devs seem to strongly reject the concept that it isnt ok to ship defective products and fix later. It will be interesting to see if it persists as incidents start to occur due to fully automated code.
Re: The short leash AI coding method for beating Fable
#135I’m not sure I understand. Babysitting models is not a multiplier IMO. If you have done 1000s of turns your harness should get sharper and less likely to go off the rails.
Also I find that on greenfield, babysitting is a must, but once you have established your house style of patterns, abstractions, and baselines, you can let any of them roam free cause they will look for examples before going forward.
I agree with the sentiment though that if you let a swarm design and code your whole codebase, you will be lost in how it fits together. More feature bloat than code bloat though from my experience
Re: The short leash AI coding method for beating Fable
#136Earlier quoted context omitted.
>>You never use “YOLO” mode (aka “dangerously skip permissions”) Do you mean this? I'm curious how are people using Claude in any way other than bypass-permissions. I've tried for so long to maintain a curated list of things Claude can use, but inevitably I would always come back only to find it stuck because it decided to pipe an output of one tool into another and that's not explicitly allowed so it stopped even th…
It does do this to frustrate you, save 30 tokens, and then waste a few thousand more when it didn't get all the context it needed by grep'ping. You have to be involved in the process though. It frequently wants to do things that are so incorrect, that even if it would be more convenient to just totally ignore it, it would be insane to actually ignore it. Do you trust it to not accidentally rm -rf the .git/ right afte…
We use perforce and Claude can't push anything to our perforce server. The worst thing it could possibly do is delete my local workspace, but that's not exactly a huge problem, would just have to sync again.
Re: The short leash AI coding method for beating Fable
#137This reminds me of the workflow I had a year ago. Miss Aider so much. Are there any good open source agents right now? Might be a good time to try one soon as Fable switches to token-based billing, which Code is designed to maximize.
Re: The short leash AI coding method for beating Fable
#138This “short leash” seems like more of a crutch to me, and a sign of not giving the AI enough detail on the problem to begin with, or not reviewing and iterating on its output. Hand-holding great models like Fable through implementation is a waste of time, and a waste of Fable. You can have increasingly nuanced discussions with stronger models, and they write a lot better code than they used to. The process of discuss…
You say you can have increasingly nuanced discussions with stronger models. What I say is, when I asked Claude why he applied a certain change I didn't understand, and boy, it was a small change, he said he "reasoned from first principles" based on the code paths. But it didn't work, and when I asked, "Okay, describe the steps of your reasoning from first principles," it literally answered that it had just made it up…
That may be true but it’s still capable of nuanced discussions.
Re: The short leash AI coding method for beating Fable
#139Earlier quoted context omitted.
You say you can have increasingly nuanced discussions with stronger models. What I say is, when I asked Claude why he applied a certain change I didn't understand, and boy, it was a small change, he said he "reasoned from first principles" based on the code paths. But it didn't work, and when I asked, "Okay, describe the steps of your reasoning from first principles," it literally answered that it had just made it up…
Posts like this are meaningless without more context - the model you're using, the harness, the initial prompt and context. Fable is better than most staff engineers at my FAANG.
But staff engineers take "responsibility"
Re: The short leash AI coding method for beating Fable
#140Earlier quoted context omitted.
It’s just parroting the current trope. Last year it was, “AI is just a stochastic parrot.” This year it’s, “AI can write the code, but a human still has to review it!” (Using AI, of course.) Give it another year and the narrative will be: “Only AI is capable of reviewing code, and only AI can review the AI’s review. Humans just need to read the AI’s final opinion so they still have meaningful oversight.” The goalpost…
Why shouldn't the goalposts move? That it was possible to beat or tie a chess master, if you had enough computational power, was basically the content of a theorem of Zermelo over a hundred years ago. It differs not a whit from tic-tac-toe. Even Eliza was practically passing the Turing test, which seems comically silly now. There's just an incredible amount of computational power so all sorts of things are possible t…
Because if the goal posts keep moving it's a sign that nobody is standing on solid ground.