Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I cancelled Claude: Token issues, declining quality, and poor support

nickyreinert.de

Post reply on HN
Print view View original

601–604 of 604 posts

Re: I cancelled Claude: Token issues, declining quality, and poor support

#601

Post by onlyrealcuzzo »

antonvs wrote: post #514
onlyrealcuzzo wrote: post #341

Earlier quoted context omitted.

> My anecdata is that it heavily depends on how much of the relevant code and instructions it can fit in the context window. Agreed, but I'm working on something >100k lines of code total (a new language and a runtime). It helps when you can implement new things as if they're green-field-ish AND THEN implement and plumb them later.

In a well-designed system, you can point an agent at a module of that system and it's perfectly capable of dealing with it. Humans also have a limited context window, and divide and conquer is always how we've dealt with it. The same approach works for agents.

> In a well-designed system, you can point an agent at a module of that system and it's perfectly capable of dealing with it.

Yes, but the problem is that LLMs don't default to well-designed systems... So, you need to aggressively stay on top of them.

Re: I cancelled Claude: Token issues, declining quality, and poor support

#602

Post by onlyrealcuzzo »

tclancy wrote: post #499
oefrha wrote: post #484

Earlier quoted context omitted.

One aspect you have to consider is the differences in human beings doing the evaluation. I had a coworker/report who would hand me obvious garbage tier code with glaring issues even in its output, and it would take multiple iterations to address very specific review comments (once, in frustration, I showed a snippet of their output to my nontechnical mom and even my mom wtf’ed and pointed out the problem unprompted);…

This is magical because you are both on the exact right path and not right. My theory is there’s a sort of skill to teasing code from AI (or maybe not and it’s alchemy all over again) and this is all new enough and we don’t have a common vocabulary for it that it’s hard for one person who is having a good experience and one person who is not to meaningfully sort out what they are doing differently. Alternatively, it…

> This is magical because you are both on the exact right path and not right. My theory is there’s a sort of skill to teasing code from AI (or maybe not and it’s alchemy all over again) and this is all new enough and we don’t have a common vocabulary for it that it’s hard for one person who is having a good experience and one person who is not to meaningfully sort out what they are doing differently.

I don't think this is a hypothesis.

Outside of asking for one-shot tasks that have been done a million times before, LLMs do not "default" to good work.

If you ask them over-and-over again to find holes in their solution, to fix them, to evaluate for tech debt, to test all cases, to re-asses after the cases if it's architecturally coherent, to compare to the closest available known good implementations, etc etc, they can eventually get what you want done unbelievably cheaply to an acceptable level of quality.

I mentioned initially - their work is unbelievably cheap, you should be EAGER to reject it. Most people wouldn't even bend down to pick a penny up off the sidewalk. They can literally pump out CLs for a penny. You shouldn't even waste time looking at "I'm done" until they've gone through 10+ rounds of reviews, refactors, bug fixes, thought of more test cases, compared to known implementations, etc.

Why are you going to spend ~$50-$100+ of your time reviewing $0.01 of LLM time?! It makes no sense!

If you just listen to them say "I'm done" and move on to their next task, it won't take too many days before you're swimming in a sea of incoherent garbage.

Re: I cancelled Claude: Token issues, declining quality, and poor support

#603

Post by Degorath »

rhubarbtree wrote: post #527
oefrha wrote: post #484

Earlier quoted context omitted.

One aspect you have to consider is the differences in human beings doing the evaluation. I had a coworker/report who would hand me obvious garbage tier code with glaring issues even in its output, and it would take multiple iterations to address very specific review comments (once, in frustration, I showed a snippet of their output to my nontechnical mom and even my mom wtf’ed and pointed out the problem unprompted);…

This is plausible. Assuming it’s true, we would see the adoption of vibe coding at a faster rate amongst inexperienced developers. I think that’s true. A counterpoint is Google saying the vast majority of their code is written by AI. The developers at Google are not inexperienced. They build complex critical systems. But it still feels odd to me, this contradiction. Yes there’s some skill to using AI but that doesn’t…

Fairly certain they do something like Anthropic does, they count the acceptance rate or something else that is fairly "optimistic" (my org has a code acceptance rate of 98,5% per the platform dashboard). So, to clarify, me accepting the suggestion and then correcting it by hand still counts as N LoC accepted.
Post reply on HN