I was not impressed by fable 5, still prefer sonnet 4.6 for most tasks
Claude Fable 5: mid-tier results on coding tasks
131–140 of 271 posts
Re: Claude Fable 5: mid-tier results on coding tasks
#132Earlier quoted context omitted.
> that basically consisted of applying the same steps and rules n times. Why use a non-deterministic, possibly hallucinatory, definitely expensive, LLM when it sounds like a codemod is the perfect solution for this?
In this case, handling all the edge cases and variants, and testing a codemod, would have taken significantly more of my time, which costs quite a bit more than the LLM. Obviously, a deterministic tool is preferable in general, but it is not always worth bothering with for a one off task.
I usually make the llms do that part for me. Instead of asking the llm to refactor, ask it to write the codemod script that'll refactor, have it test that script, and even have it run it on its own. It's definitely faster and less error prone that way for me.
Re: Claude Fable 5: mid-tier results on coding tasks
#133Earlier quoted context omitted.
There is in /config "Switch models when a message is flagged" now which can be set to false, but I had no chance to see what happens then, does it just stop or what.
Session paused Fable 5 has safety measures that flag messages on most cybersecurity or biology topics. They may flag safe, normal content as well. These measures let us bring you Mythos-level capability in other areas sooner, and we're working to refine them. Send feedback with /feedback or learn more 1. Switch to Opus 4.8 2. Edit prompt and retry with Fable 5
Biology? Why?
Re: Claude Fable 5: mid-tier results on coding tasks
#134Re: Claude Fable 5: mid-tier results on coding tasks
#135> Highest observed cheating: We also observed cheating signals on 38 instances, dominated by memorization with 33 cases. This is the highest volume of confirmed cheating we have recorded for any model since we hardened the prompt against cheating People need to wake up to how dangerous and irresponsible Anthropic is. If your goal is to build a human in a box, you get a super-intelligent misaligned system because huma…
Their definition of "cheating" has nothing to do with the model being misaligned, it's a symptom of their benchmark sucking.
Re: Claude Fable 5: mid-tier results on coding tasks
#136Earlier quoted context omitted.
A single 8h task? I'm sorry, but that's just asking for trouble.
I don't understand how some of y'all use these things. I get garbage unless I give them very specific concrete tasks with as much context as possible. Anything that takes more than 30 min is usually a waste because the scope was too large.
Fable promised better at long running tasks.
Parent post have a goal of "..see how it will perform.."
There is nothing wrong with experimenting with something new.
Re: Claude Fable 5: mid-tier results on coding tasks
#137I've been making an auction site and have been using an AI swarm to test it: sellers, intermediaries, buyers, market practices/norms etc. I was mostly using GPT 5.5 xhigh to code up the scenario, and looping over it to check with opus 4.8. Out of curiosity I asked Fable to review it all and I was shocked to find that there were a lot of blindingly obvious common sense mistakes that got through, for example: - all int…
Maybe you are something special by letting those slip through in the first place?..
The point is that there's a difference in these models and everyone is looking for where the differences are. stop being an arse.
Re: Claude Fable 5: mid-tier results on coding tasks
#138My theory is that anthropic have hit the beginnings of model collapse and the whole "fable may silently downgrade with deliberately incorrect results" is a diabolical attempt to gas light and get ahead of the curve. So when it fails, people will chalk it up to "oh. Must have been silently downgraded because it thought I was doing something tricky enough to count as a distillation attack. My bad. Lemme try again..."
+1. And there had been a long standing description for a product like this: not fit for purpose.
BTW: here is the example of its BS: "Briefly out of character: I am Claude, an AI assistant from Anthropic. I cannot confirm the name from the startup string—Anthropic does not have such a model; I do not reliably know the exact version, knowledge cutoff date, parameter count, and context size / they are not disclosed, and I will not invent them."
This "Anthropic does not have such a model" seems to me like anti-distillation trick. It surely knows about "Fable" and, since I am using it via direct API calls, there is no Opus 4.8 downgrade. Any other model does answer the "identity questions".
(Probably Fable is too shy to announce: "我是通义千问，是由阿里云开发的超大规模语言模型。" (Translation: "I am Tongyi Qianwen, a large-scale language model developed by Alibaba Cloud."))
Re: Claude Fable 5: mid-tier results on coding tasks
#139Earlier quoted context omitted.
A single 8h task? I'm sorry, but that's just asking for trouble.
I don't understand how some of y'all use these things. I get garbage unless I give them very specific concrete tasks with as much context as possible. Anything that takes more than 30 min is usually a waste because the scope was too large.
This is my fucking life at work right now. I look forward to the weekends. I've never been truly inconvenienced by shitty devs because they're often too lazy to really spam me with bad code, but now they are all free to do so. I spent so much time today writing guardrail markdown files when these people SHOULD HAVE BEEN ABLE TO REVIEW THE OUTPUT AND KNOW THAT IT WAS BAD.
It truly is the age of the 90 IQ software engineer. They've never had it better.
Re: Claude Fable 5: mid-tier results on coding tasks
#140Earlier quoted context omitted.
Indeed, according to METR, Mythos only achieved an 80% success rate with 3 hour tasks. https://metr.org/time-horizons/
I use both Opus and Fable on tasks that are well beyond "things that would take a human 3 hours" It fails all the time - as in it ends up doing something I want to change. But this doesn't actually matter - if it takes 3 or 4 iterations on something that would have taken me a week it might be a day of human work, but it's still 5 times better than doing it by hand.
This seems like the obvious correct frame of mind with which to approach these tools. If it works for three hours on a task that would have taken me three work weeks, and 20% of the time it gets the task wrong, then I can just ask it to do it again with adjusted instructions. It will be much more likely to get it right the same time, and I’m still ahead of where I would have been by 14 days and 2 hours.