It definitely felt less capable recently, I thought I was imagining it, but it was noticeably more difficult to get it to help on tasks that usually aren't so hard.
Claude Code daily benchmarks for degradation tracking
351–360 of 372 posts
Re: Claude Code daily benchmarks for degradation tracking
#352Earlier quoted context omitted.
How about how Claude 2.1.x is "literally unusable" because it frequently completely hangs (requires kill -9) and uses 100% cpu? https://github.com/anthropics/claude-code/issues/18532
What OS? Does this happen randomly, after long sessions, after context compression? Do you have any plugins / mcp servers running? I used to have this same issue almost every session that lasted longer than 30 minutes. It seemed to be related to Claude having issues with large context windows. It stopped happening maybe a month ago but then I had it happen again last week. I realized it was due to a third-party mcp s…
MacOS; no mcp; clear context; reliably reproducible when asking claude review a pr with a big VCR cassette.
Re: Claude Code daily benchmarks for degradation tracking
#353Earlier quoted context omitted.
If you don't do it that way then resizing the terminal corrupts what's on screen.
It's possible to handle resizes without all this machinery, most simply by clearing the screen and redrawing everything when a resize occurs. Some TUI libraries will automatically do this for you. Programs like top, emacs, tmux, etc are most definitely not implemented using this stack, yet they handle resizing just fine.
That doesn't work if you want to preserve scrollback behavior, I think. It only works if you treat the terminal as a grid of characters rather than a width-elastic column into which you pour information from the top.
Re: Claude Code daily benchmarks for degradation tracking
#354Earlier quoted context omitted.
https://news.ycombinator.com/item?id=46819744
I'm not saying CC doesn't have issues and curious design decisions - but your terminal should only be rendering (at most) a single window of characters every frame no matter what. CC shouldn't be capable of making that take 10% of a modern GPU regardless of what CC does.
¯\_(ツ)_/¯ just vscode plus claude in the terminal on win10.
Re: Claude Code daily benchmarks for degradation tracking
#355Earlier quoted context omitted.
The primary (non malicious, non stupid) explanation given here is batching. But I think you would find looking at large-scale inference the batch sizes being ran on any given rig are fairly static - there is a sweet spot for any given model part ran individually between memory consumption and GPU utilization, and generally GPUs do badly at job parallelism. I think the more likely explanation is again with the extreme…
Why do you think batching has anything to do with the model getting dumber? Do you know what batching means?
Well if you were to read the link you might just find out! Today is your chance to be less dumb than the model!
Re: Claude Code daily benchmarks for degradation tracking
#356Earlier quoted context omitted.
You are funny. Anthropic refuses to issue refunds, even when they break things. I had an API token set via an env var on my shell, and claude code changed to read that env var. I had a $10 limit set on it, so found out it was using the API, instead of my subscription, when it stopped working. I filed a ticket and they refused to refund me, even though it was a breaking change with claude code.
Anthropic just reduced the price of the team plan and refunded us on the prior invoice. YMMV
So they have no durable principles for deciding who or what to refund… doesnt that make them look even worse…?
Re: Claude Code daily benchmarks for degradation tracking
#357Earlier quoted context omitted.
Why do you think batching has anything to do with the model getting dumber? Do you know what batching means?
Well if you were to read the link you might just find out! Today is your chance to be less dumb than the model!
I checked the link, it never says that the model's prediction get lower quality due to batching, just nondeterministic. I don't understand why people conflate these things. Also it's unlikely that they use smaller batch sizes when load is lower. They just likely spin up and down GPU serves based on demand, or more likely, reallocate servers and gpus between different roles and tasks.
Re: Claude Code daily benchmarks for degradation tracking
#358Earlier quoted context omitted.
He doesn't: https://x.com/trq212/status/2014051501786931427
For those who don't want to visit X: Most people's mental model of Claude Code is that "it's just a TUI" but it should really be closer to "a small game engine". For each frame our pipeline constructs a scene graph with React then -> layouts elements -> rasterizes them to a 2d screen -> diffs that against the previous screen -> finally uses the diff to generate ANSI sequences to draw We have a ~16ms frame budget so w…
Yes yes I'm familiar with the tweet. Nonetheless they drop frames all the time and flicker frequently. The tweet itself is ridiculous when counterpoints like Vim exist, which is much higher performance with much greater complexity. They don't even write much of what the tweet is claiming. They just use Ink, which is an open-source rendering lib on top of Yoga, which is an open-source Flexbox implementation from Meta.
Re: Claude Code daily benchmarks for degradation tracking
#359Earlier quoted context omitted.
Interesting. On first glance that seems over engineered. I wonder what the reason is for doing it that way?
If you don't do it that way then resizing the terminal corrupts what's on screen.
Counterpoint: Vim has existed for decades and does not use a bloated React rendering pipeline, and doesn't corrupt everything when it gets resized, and is much more full featured from a UI standpoint than Claude Code which is a textbox, and hits 60fps without breaking a sweat unlike Claude Code which drops frames constantly when typing small amounts of text.
Re: Claude Code daily benchmarks for degradation tracking
#360Im using Claude daily. Mostly delegating boring stuff I can do myself but its a waste of my time now.
I store my prompts, so I know I often run the same task multiple times over weeks span.
After working with it for pas half a year I have to say the quality pf responses is steadily going down.
Feels like cost optimizations. Overall the worse it performs the more stuff I have to do myself, because I won’t waste time tweaking instructions every time it happens. It wpulf waste too much of that time.
So seems we are swinging back the pendulum.