Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The short leash AI coding method for beating Fable

blog.okturtles.org

Post reply on HN
Print view View original

21–30 of 268 posts

Re: The short leash AI coding method for beating Fable

#21

Post by reinitctxoffset »

hungryhobbit wrote: post #12

I I mean, it's like writing a book about how to use React or Django or some other major software ... after you used it for one project for a month! Authors: I know this is the Internet, and I know bloggers blog about whatever pops into their head ... but if you are going to act like an authority, how about you learn more than the average reader before you start telling them authoritatively what to do?

It's an open problem of clearly large value how to get reliably useful and trustworthy outcomes from AI systems in many domains, software is maybe the signal example of that. If one had solved it resoundingly and scaleably, one could in fact "get rich quick".

It is unsurprising that a lot of people claim to know how to get rich quick.

I believe it is possible to solve this problem, and I have my own horses in the race which I won't threadjack to promote here, but it's the central problem of our profession at the moment. We've all seen the truly discontinuous outcomes and we've all seen allegedly national security dangerous models (which at one time was GPT-3) faceplant with it's shoelaces tied together. I wanted to see if Fable was really all that and I left it overnight on some fairly straightforward C++ (code DSv4 Flash works on with moderate supervision) and it's pretty roast worthy, I gave it a chance to redeem itself this morning and it's ticked up a bit (I still think it's roughly Opus 4.8 with a Project Zero fine tune and DRO trained off the constant gratuitous yield tic which is pretty clearly an intentional gimp).

I give all such claims 30 seconds of my time because someone is going to actually be right one of these days.

Re: The short leash AI coding method for beating Fable

#22

Post by hungryhobbit »

kristianc wrote: post #15
hungryhobbit wrote: post #12

I I mean, it's like writing a book about how to use React or Django or some other major software ... after you used it for one project for a month! Authors: I know this is the Internet, and I know bloggers blog about whatever pops into their head ... but if you are going to act like an authority, how about you learn more than the average reader before you start telling them authoritatively what to do?

People are doing what they've always done with any other new technology, and sharing what, personally, works for them. People can take or leave the advice.

Right but there's a marked difference between a "I just tried this new tech and here's what I think" vs. "I've used this tech for a few months and now I'm going to speak like I know everything about it".

I have no beef with people writing about new tech, but I do have beef with claiming that "____ is the correct way to do it" ... based on nothing except "I feel proud of the last three months I spent with Claude".

Re: The short leash AI coding method for beating Fable

#23

Post by sothatsit »

This “short leash” seems like more of a crutch to me, and a sign of not giving the AI enough detail on the problem to begin with, or not reviewing and iterating on its output.

Hand-holding great models like Fable through implementation is a waste of time, and a waste of Fable. You can have increasingly nuanced discussions with stronger models, and they write a lot better code than they used to. The process of discussing designs and their implementations, questioning things that look weird to you, and actually reading the AI’s responses also helps to find better solutions.

For example, one time I wanted to write a greedy solver for a problem, and in my discussion with Opus on the idea it suggested using an existing MILP library to solve the problem exactly. I’d never even heard of MILP, but my final implementation ended up being better and simpler than what I’d have done alone.

Re: The short leash AI coding method for beating Fable

#24

Post by taormina »

gambiting wrote: post #13
jonplackett wrote: post #6

I thought this was how everyone who can actually code uses AI for anything that’s actually important. Am I wrong? Are you guys just YOLOing everything these days?

>>You never use “YOLO” mode (aka “dangerously skip permissions”) Do you mean this? I'm curious how are people using Claude in any way other than bypass-permissions. I've tried for so long to maintain a curated list of things Claude can use, but inevitably I would always come back only to find it stuck because it decided to pipe an output of one tool into another and that's not explicitly allowed so it stopped even th…

It does do this to frustrate you, save 30 tokens, and then waste a few thousand more when it didn't get all the context it needed by grep'ping. You have to be involved in the process though. It frequently wants to do things that are so incorrect, that even if it would be more convenient to just totally ignore it, it would be insane to actually ignore it. Do you trust it to not accidentally rm -rf the .git/ right after it helpfully force pushes to remote? I don't. Even if I don't expect it to do that, why would I ALLOW it to be able to?

Re: The short leash AI coding method for beating Fable

#25

Post by WhitneyLand »

This post seems like some decent advice mixed in with a lot of overconfidence and unverifiable claims.

“expert developers whose skills have reached the point where they outclass any and all “frontier AI models” in their area of expertise”

Are any developers saying they outclass any and all frontier models? I’d say at best it’s mixed at this point. The best developers still do certain things better, but not even close to all things.

“The problem is that even code written and/or reviewed by Fable 5, will stink”

I’m skeptical. Example prompt and output please.

Re: The short leash AI coding method for beating Fable

#26

Post by reinitctxoffset »

threethirtytwo wrote: post #4
sscaryterry wrote: post #2

There really wasn't much substance to this article.

It’s just parroting the current trope. Last year it was, “AI is just a stochastic parrot.” This year it’s, “AI can write the code, but a human still has to review it!” (Using AI, of course.) Give it another year and the narrative will be: “Only AI is capable of reviewing code, and only AI can review the AI’s review. Humans just need to read the AI’s final opinion so they still have meaningful oversight.” The goalpost…

The regress ends somewhere, because (barring some pretty sharp changes to the way the law works basically everywhere) ultimately someone has to certify the outcomes as acceptable. This might be in the form of the market (though AI-adjacent stuff seems extremely prone to prolonged market failures), this might be regulatory in nature. This might be the executive management of the companies involved.

Personally I think that if you cranked the capability up high enough the first person you'd run into who absolutely demanded more than vibes and didn't care about your singularity thesis would be the representative of a reinsurance firm: mostly to do serious stuff without bending the law, you need insurance, and I am unaware of anyone writing serious policies (certainly not ones that make any economic sense) that underwrite the risk of AI autonomy outcomes financially.

When Swiss Re writes a policy that Anthropic Cinematic Universe or whatever iteration we're on won't fuck it up?

Now maybe we're talking. Until then you ask three practitioners and get nine answers, no one knows what they're talking about unless they're doing a really good job keeping it quiet (and that's probably what you'd do!).

Re: The short leash AI coding method for beating Fable

#27

Post by densekernel »

sothatsit wrote: post #23

This “short leash” seems like more of a crutch to me, and a sign of not giving the AI enough detail on the problem to begin with, or not reviewing and iterating on its output. Hand-holding great models like Fable through implementation is a waste of time, and a waste of Fable. You can have increasingly nuanced discussions with stronger models, and they write a lot better code than they used to. The process of discuss…

I tend to agree,

If you have invested significantly in the planning phase and there is momentum in the architecture and conventions that already exist in the project, the implementation phase might not need as much oversight as is suggested here.

> You can discover that your initial idea was dumb and a better one exists

The planning and architecture phase is usually where I make these types of discovery at a high level.

> Your agent might go “off the rails” and start doing something you don’t want it to do

Candidly these orthogonal, inadvertent edits aren't as bad as they once were and for impactful changes there should be at least some test coverage, even if that test coverage is just "freezing" what was implemented.

As you mentioned the final review discussion is a good chance to verify beyond what review or adversarial review agents find.

Re: The short leash AI coding method for beating Fable

#28

Post by arcanemachiner »

taormina wrote: post #24
gambiting wrote: post #13

Earlier quoted context omitted.

>>You never use “YOLO” mode (aka “dangerously skip permissions”) Do you mean this? I'm curious how are people using Claude in any way other than bypass-permissions. I've tried for so long to maintain a curated list of things Claude can use, but inevitably I would always come back only to find it stuck because it decided to pipe an output of one tool into another and that's not explicitly allowed so it stopped even th…

It does do this to frustrate you, save 30 tokens, and then waste a few thousand more when it didn't get all the context it needed by grep'ping. You have to be involved in the process though. It frequently wants to do things that are so incorrect, that even if it would be more convenient to just totally ignore it, it would be insane to actually ignore it. Do you trust it to not accidentally rm -rf the .git/ right afte…

Don't give it the necessary permissions to push to remote.

I run mine in a container, so it doesn't have access to the SSH key I use to push.

Re: The short leash AI coding method for beating Fable

#29

Post by semiquaver »

moezd wrote: post #18

LLMs are still next token predictors, just because you can give it more vague instructions and it still finds the right steps to follow, it doesn't mean it's intelligent. It means you're speaking the same language as the harness they trained your model on. And that has a limit. If you are stuck at PoC level or simple apps, you have no idea how limited the current models still are. There you really need to break tasks…

Yeah, and you’re just a next-word-sayer.

Re: The short leash AI coding method for beating Fable

#30

Post by afro88 »

Maybe I'm too optimistic, but given appropriate skills and references (not just for writing but also reviewing) and intelligent use of subagents for isolated reviews and checks, you can lengthen the leash a bit.

But you still need to properly review plans and PRs to keep a good mental model of the codebase. This effectively limits the number of tasks being done in parallel to maybe 2-3. Though you'll be mentally exhausted and probably start to make mistakes or take shortcuts in reviews yourself.

Post reply on HN