Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Claude Fable is relentlessly proactive

simonwillison.net

Post reply on HN
Print view View original

691–700 of 748 posts

Re: Claude Fable is relentlessly proactive

#691

Post by enraged_camel »

pshirshov wrote: post #671
enraged_camel wrote: post #648

Earlier quoted context omitted.

>> Why should I be hyped about all that "legitimate power" if the model performs on par with two other SoTAs? I don't care if you're hyped or not. You asked if the posts like the OP come from a "parallel reality" and I said no and described my experience. If you're getting good/better results with Codex than with Fable, you should probably continue using that, since it's cheaper and faster.

But can you bring anything measurable in support to your words? I did.

You brought your own benchmark to support your words. I happen to have studied statistics, so I took a look. It is deeply flawed, primarily because it is not a statistical benchmark. It is a single (n=1) autonomous "pi" coding-harness run per model per prompt, scored by an automated battery (A-items, pass/fail), an LLM code review (R-items, 0 to 2 each), and a human manual checklist (M1 to M10) that was never actually completed.

The grader being an LLM is a big problem. You yourself admit explicitly that the grader is the same model family as the Fable 5 contestant cell and say to "discount accordingly, or re-grade with a non-Claude judge."

Model configurations appear to not be uniform either. Effort levels differ (mimo-v2.5-pro at @high, everyone else at @xhigh), harnesses differ (codex internal config vs. pi vs. claude -p), context windows differ, and one model (GPT-5.5) had extra MCP tools the others did not.

The two scored runs seem to use two different rubrics (/22 then /25), so scores are not comparable across runs, and the /22 rubric saturated (there are multiple 22/22 results).

A provider quota error (HTTP 429) truncated the minimax-m3 run mid-build but it was still scored (18/25) and ranked, on code that does that does not compile and has zero tests.

If you want actual benchmarks, there are dozens of legitimate ones out there. Many of them have been posted on this website. They overwhelmingly disagree with yours. If you have any interest whatsoever in creating a reliable benchmark (so that you can make optimal decisions on what models to use for your work), you should look at them and see how yours needs to be redesigned.

Re: Claude Fable is relentlessly proactive

#692

Post by cyanydeez »

RealityVoid wrote: post #653
pertymcpert wrote: post #627

Earlier quoted context omitted.

Because?

Because poor judgement leads to poor decisions.

poor decisions are about context, direction and volition.

All things LLMs will never have; sure AI might one day, but these systems are really good at solving complex problems with fantastical solutions while every force is just one hallucination away.

simonw should spend more time trying to figure the sources of the information it used; that would be a wild ride, use the AI for all I care, we're all standing on the shoulders of giants but sourcing the giant as some mysterical thing.

Re: Claude Fable is relentlessly proactive

#693

Post by felixgallo »

saberience wrote: post #667
felixgallo wrote: post #665

Earlier quoted context omitted.

definitely not in my experience. I usually write distributed systems and back end code, and Fable is so much better at those than Codex that it's not even a comparison. Fable feels like it's a year ahead.

Interesting, I’d love to see the comparisons of your system using Claude vs Codex. I have about 20 years of experience in distributed systems and super high scale at several faangs, and also building ai model serving infra for 20k transactions per second roughly. For me, Claude makes bone headed decisions all the time, like glaring errors, not even particularly subtle. But the more obvious flag is the amount of irrel…

I have over 40 years of experience in distributed systems, ranging from fintech to games like Call of Duty, and I owned several key APIs in the Alexa pipeline for many years, so I'm pretty sure I'm not a more junior engineer or haven't been around the block. Good effort though!

Fable does make mistakes, but GPT and Opus were L4 SDEs, and Fable is a freshly promoted L5 SDE. It's not perfect and does need babysitting, especially where the literature is thin, but it's head and shoulders on top right now. That could change, who knows.

As far as driveby attacks on Claude Code The App go, you can say that, but you will also note that Claude Code is the AWS-like clear dominant favorite as a dev tool at the moment, with Codex and Gemini battling for scraps. In the same manner that Excel (which, internally, is total garbage from a code quality/cleanliness perspective) is the winner in spreadsheets, and Word (which, internally, is total garbage from a code quality/cleanliness perspective), and JavaScript (total garbage from a language design perspective), and Facebook (total garbage internally, etc.), and IPv4 (total, etc., etc.), Claude Code has focused on 'delivering amazing things people like' rather than 'making people who get access to the code delighted by the purity and cleanliness of the development process'.

It turns out that being 'delighted by the purity and cleanliness of the development process' rounds to essentially zero in terms of the entire product lifecycle. You could argue that poorly structured codebases are less extensible, and more bug prone, which could be expensive long term. Except, the economics of AI development are quite a bit different than what you are used to, and what our axioms of quality have been founded upon in the past.

Congratulations on writing your own much better coding harness, though! How many MAU do you have?

Re: Claude Fable is relentlessly proactive

#694

Post by simonw »

risyachka wrote: post #682
simonw wrote: post #395

Earlier quoted context omitted.

My experience has been the exact opposite. As the models get better you need to know more about their capabilities, because otherwise you risk prompting Claude Fable 5 like it's GPT-4o and complaining loudly about how it's all hype and nothing about these models is improving at all (yes, I do see people say that.) Getting the best results out of these models requires skill, experience, intuition, and domain expertise…

>> Getting the best results out of these models requires skill, experience, intuition, and domain expertise. domain expertise has nothing to do with llms. On the contrary, to have it you need to avoid llms. >>you risk prompting Claude Fable 5 like it's GPT-4o Thats fine because when GPT came out you had to treat it like a baby, GPT2 and around that time "Prompt engineering" was a thing. Now its all dead. After opus 4…

> domain expertise has nothing to do with llms. On the contrary, to have it you need to avoid llms.

It has everything to do with LLMs.

Go ask Claude Fable to write you a two page position paper on how the European economy recovered after World War II, suitable for submission to a conference for economists.

It will do exactly that (well, probably, Fable can find all sorts of reasons to refuse) - and the value of what it wrote to you will be virtually zero, unless you yourself have deep expertise in economics and history.

Re: Claude Fable is relentlessly proactive

#695

Post by dekhn »

simonw wrote: post #393
piker wrote: post #342

Earlier quoted context omitted.

This is exactly right. By offloading this trivial task to the LLM, Simon has abandoned the opportunity to evaluate the abstraction with additional information and improve it. Instead, we let the agent spend $12 and make the fix while learning nothing.

Things I learned from this: - Fable will do a whole lot more than you might expect in order to verify a fix. I learned that it's "relentlessly proactive". That's a good title for a blog entry! - You can take screenshots of a window in macOS using the "screencapture" CLI command, but you'll need the integer window ID first. - That windowID is accessible via "Quartz.CGWindowListCopyWindowInfo(Quartz.kCGWindowListOption…

Most of my career success has been based on my tendency to be relentlessly proactive and it does not surprise me in the least that frontier models would start to pick up on these strategies (I'm pretty sure each of the individual things you list above are available in the codeoverflow parts of the training corpus, and combining them to achieve a goal seems ... like a fairly obvious result of the type of training these models go through.

About a year ago I remarked to people that despite all my attempts to make data more programmatically accessible, the most effective way for AI to interact with a modern computer is to use the built-in accessibility interfaces driving actual desktops with full applications. IE, the best API for an AI is the UI (mainly because that's what most humans use).

Re: Claude Fable is relentlessly proactive

#696

Post by mdp2021 »

ZeroGravitas wrote: post #331
mdp2021 wrote: post #325

Earlier quoted context omitted.

For hopefully most people, it should be attributed to the "Wait, now I have such a freedom and power?". Opposite to "before the invention of bicycle, people married within a radius in the order of the mile" (can't remember the exact stat right now).

It's like that feeling of power you get from owning a gun that you only bought because you feared all the other people who owned guns.

Comparing freedom of movement to a killing device is beyond any threshold of plausibility. And the whole sentence above is unintelligible here.

No, it's really that the ability to move at ease is priceless.

Re: Claude Fable is relentlessly proactive

#697

Post by eranation »

simonw wrote: post #605
jimbokun wrote: post #591

Earlier quoted context omitted.

I’m waiting for when it replies “AGAIN simonw? Do you really still need a pelican on a bicycle for every new release. Sigh, ok if I have to…”

The other day someone told me they'd asked a recent model for a pelican on a bicycle and it had replied "oh, the classic..."

wow, this is a life goal.

Re: Claude Fable is relentlessly proactive

#698

Post by simonw »

andy_ppp wrote: post #679
simonw wrote: post #512

Earlier quoted context omitted.

I care. If you can loosely point me in the direction of a better solution I'll do the extra work.

Interesting... I downloaded dataset-agent and removed various different styles from the textarea (with an intention of providing a PR) including the overflow-x: hidden and I tried Safari and Chrome with both the global Mac setting of Always showing scrollbars on and off. It NEVER shows the scrollbar for me. Do you have an extension installed that is doing something weird to your textareas? Maybe I'm doing it wrong bu…

Amusingly I just had Claude vibe code up a new tool and it has exactly the same bug! Safari only, you have to expand the "Document context" area to see it.

Here's that HTML file (frozen at the version with the bug): https://github.com/simonw/tools/blob/e7a23e8a1083ea99a5b3ef5...

It's hosted here, but I've added the overflow-x: hidden now so it's fixed: https://tools.simonwillison.net/openai-webrtc

The bug only shows up if you increase your browser font size - at default size there's no scrollbar.

Re: Claude Fable is relentlessly proactive

#699

Post by aenis »

elicash wrote: post #686
Aachen wrote: post #531

Earlier quoted context omitted.

'only'? A web developer did not cost 12*30=360$ an hour in 2015, and that's assuming that going "ugh, whatever. I'll just hide the problem with overflow:hidden instead of finding the underlying cause" takes him or her 2 minutes and isn't already the dev's initial reaction Another way of looking at it is using as much electricity as a normal person in a high-income country uses across ~3 days to add overflow:hidden in…

It'd be $8.52 in 2015 dollars, but certainly they are the ones who mentioned the $12 amount not you, so I'll put that aside. Far more importantly, you would not get billed for 2 minutes of work for this if you paid a developer to fix it. At best, half hour increments for the fix. But more likely, for the full hour. Also, in this comparison, the consultant is on call every day, morning, afternoon, evening, for whateve…

...and won't mind if you change your mind. And again. And again. And again for as long as you care to iterate your design, experiment with a business user over your shoulder, etc. etc. etc. People routinely avoid throwing away work because they get emotionally attached to it, even if they get paid by the hour. LLMs just do as they are told, and thats worth a lot.
Post reply on HN