101–110 of 131 posts
Earlier quoted context omitted.
“Coding” can be more like communing with an otherworldly presence via esoteric gesturing nowadays than ever. Superstition leads countries and companies.
Every day we get closer to being mystics having to commune with orbs in order to create phenomena.
Earlier quoted context omitted.
I feel like a conspiracy theorist but it feels like every new model release from both Anthropic and OpenAI has 1-2 weeks of fantastic performance then a gradual (and sometimes not so gradual) decline in intelligence. It's like they're quantising the model in the background to optimise for available compute/RAM. Which, if I put my MBA hat on, would make perfect sense. Why not halve the required RAM for "90%" of the pe…
And yet: https://marginlab.ai/trackers/claude-code-historical-perform... There's clearly random variation, but it also shows each model release is just genuinely better. ( With the exception of a heavily degraded week of opus 4.7, which was acknowledged as a problem at the time. ) There's a psychology of getting used to models after being wowed by the new performance. It sets in as your new baseline expectations, and…
Earlier quoted context omitted.
“Coding” can be more like communing with an otherworldly presence via esoteric gesturing nowadays than ever. Superstition leads countries and companies.
Every day we get closer to being mystics having to commune with orbs in order to create phenomena.
This matches my experience of Opus 5 being a nice improvement over Opus 4.8, but not being revolutionary like Fable felt. I’ve now replaced my use of Opus 4.8 xhigh with Opus 5 medium, and I’m using less tokens and it’s quicker. I can understand people being annoyed by its writing style but for getting work done that really doesn’t bother me. I’ve been really enjoying using it.
Can you elaborate on what felt revolutionary to you about Fable?
The code still took weeks to clean up, but it worked and was correct. It felt then, and still feels, like a big step change on very hard problems. These are problems I would previously expect to take a month or longer to implement.
I have also noticed Fable can handle much more nuance when reasoning through writing and research, but that is harder to quantify.
Earlier quoted context omitted.
Can you elaborate on what felt revolutionary to you about Fable?
I got Fable to run overnight and I woke up to a working prototype of a very complex feature. And then I did it again for another complex feature the next night. The code still took weeks to clean up, but it worked and was correct. It felt then, and still feels, like a big step change on very hard problems. These are problems I would previously expect to take a month or longer to implement. I have also noticed Fable c…
SCB is definitely an underrated benchmark. For me the unique selling point is that it more closely mirrors software development by not stopping after a single task. The agent has to keep code clean. The only disadvantage is all the problems are greenfield and not git inited so the agents don’t make use of git diffs. I’ve used SCB as part of my assessment of agent skills (superpowers, GSD etc) https://orcabot.com/labs…
I hope the big labs will start using this benchmark in their RL pipelines. Reducing complexity in generated code should be the number 1 priority, in my opinion. The holy grail for me is models implementing features while reducing LoC (i.e., choosing the right abstractions). What is also nice about this benchmark is that it can be used to iterate on prompts/skills for reducing code complexity.
It's just as easy to go too far in the direction of shoving too much logic on a single line in the name of reducing lines of code.