181–190 of 372 posts
Earlier quoted context omitted.
I guess I just don't know how to square that with my actual experiences then. I've seen sporadic drops in reasoning skills that made me feel like it was January 2025, not 2026 ... inconsistent.
LLMs sample the next token from a conditional probability distribution, the hope is that dumb sequences are less probable but they will just happen naturally.
I have yet to experience any degradation in coding tasks I use to evaluate Opus 4.5, but I did see a rather strange and reproducible worsening in prompt adherence as part of none coding tasks since the third week of January. Very simple queries, even those easily answered via regular web searching, have begun to consistently not result accurate results with Opus 4.5, despite the same prompts previously yielding accur…
I've noticed a degradation in Opus 4.5, also with Gemini-3-Pro. For me, it was a sudden rapid decline in adherence to specs in Claude Code. On an internal benchmark we developed, Gemini-3-Pro also dramatically declined. Going from being clearly beyond every other model (as benchmarks would lead you to believe) to being quite mediocre. Delivering mediocre results in chat queries and coding also missing the mark. I did…
Claude-Code is terrible with context compaction. This solves that problem for me.
Earlier quoted context omitted.
Floating point math isn't associative for operations that are associative in normal math.
That would just add up to statistical noise instead of 10% degradation over a week.
Earlier quoted context omitted.
I guess I just don't know how to square that with my actual experiences then. I've seen sporadic drops in reasoning skills that made me feel like it was January 2025, not 2026 ... inconsistent.
LLMs sample the next token from a conditional probability distribution, the hope is that dumb sequences are less probable but they will just happen naturally.
"No no yeah bro no I'm good like really the work's done and all yeah sorry I missed that let me fix it"
[SWE-bench co-author here] It seems like they run this test on a subset of 50 tasks, and that they only run the test once per day. So a lot of the movement in accuracy could be attributed to that. I would run on 300 tasks and I'd run the test suite 5 or 10 times per day and average that score. Lots of variance in the score can come from random stuff like even Anthropic's servers being overloaded.
https://www.anthropic.com/engineering/a-postmortem-of-three-...
[SWE-bench co-author here] It seems like they run this test on a subset of 50 tasks, and that they only run the test once per day. So a lot of the movement in accuracy could be attributed to that. I would run on 300 tasks and I'd run the test suite 5 or 10 times per day and average that score. Lots of variance in the score can come from random stuff like even Anthropic's servers being overloaded.
According to Anthropic: "We never reduce model quality due to demand, time of day, or server load." https://www.anthropic.com/engineering/a-postmortem-of-three-...
You have to test inter-day variation. Many have noticed a sudden drop off at certain times.