Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I benchmarked Claude Code's caveman plugin against "be brief."

maxtaylor.me

Post reply on HN
Print view View original

41–50 of 71 posts

Re: I benchmarked Claude Code's caveman plugin against "be brief."

#41

Post by magicalhippo »

ramesh31 wrote: post #29
max-t-dev wrote: post #16

Earlier quoted context omitted.

Caveman doesn't compress the reasoning, only the output. The model still does its full reasoning before generating the response, caveman just affects how the final response is formatted.

>The model still does its full reasoning before generating the response, caveman just affects how the final response is formatted Right, and that final response forms the latest context for your next follow-up prompt. Not having that final reasoning laid out in the conversation history leaves a huge gap in successive reasoning. I remember playing around with this idea in the Sonnet 3.x days and it was immediately obv…

I don't know how Claude and such do it, but latest Qwen model supports preserving reasoning between calls, which based on what I heard does help a fair bit.

Re: I benchmarked Claude Code's caveman plugin against "be brief."

#42

Post by gavmor »

lofaszvanitt wrote: post #3

Caveman is useless for me. We are in the year 2026, computers are here to serve me, and bring me comfort. Caveman is a caveman, speaks like an idiot. I don't want to interact with an idiot. It's irritating, and as the article states, an overhyped turd. It is the same idiocy that permeates EV cars. You buy an expensive car to go from A to B and at the same time offer you comfort. When I have to think about using the s…

Doesn't "be brief" lobotomize the model, too? The good stuff comes at the ends of difficult sentences, ie the latent gold lies at the end of fully arcing latent rainbows, no?

Re: I benchmarked Claude Code's caveman plugin against "be brief."

#43

Post by Aurornis »

I still can’t believe that people take Caveman seriously.

It’s a funny joke, but saving a couple hundred tokens in the final output is going to be negligible, especially when coding where it’s common to go through hundreds of thousands of tokens in a session. You also have to consider the additional tokens consumed by the skill itself (acknowledging that output tokens are billed at a different rate).

I got a kick out of it when it was released, but now that I’m seeing it repeated as a useful operation it’s apparent how much cargo culting is going on in this space.

Re: I benchmarked Claude Code's caveman plugin against "be brief."

#44

Post by dataviz1000 »

max-t-dev wrote: post #9
dataviz1000 wrote: post #6

Earlier quoted context omitted.

> there was 1 run per prompt per arm My understanding is that there was only 1 run per configuration? If that is correct, because of the run-to-run variability, it really doesn't say much. It will take several trails per prompt per arm before it will look like it is stabilizing on a plot. It is prohibitively expensive so I've been running same prompt, same model 5 times in order to get a visual understanding of perfo…

Yeah fair point. The benchmark is single-run per arm-prompt pair, so the variance finding on safety categories could be noise rather than signal. Findings doc flags this for the score deltas (anything under 0.02 between arms is in the judge's noise floor) but I should have applied the same caveat to the per-question token variance, which I didn't. Will read the lambda variance write-up. Multi-trial with cost classifi…

I'm trying to wrap my mind around this. Anything you explore and share is awesome. Thanks for the blog post.

If you want to test it across coding tasks, have a look at https://github.com/adam-s/testing-claude-agent

Re: I benchmarked Claude Code's caveman plugin against "be brief."

#46

Post by refactor_master »

Can someone give me a sound argument for why, when these things supposedly hold:

- LLMs scale with amount of data on the subject

- Even frontier labs themselves have a hard time gauging exactly how well-performing models are, across a quite rigorous set of tests in all aspects

then, how can this be true:

Using a low-data "niche language" (what is the volume of literature written in Caveman?) is supposedly of equal performance, when this anecdotally doesn't hold for e.g. niche code languages, proven by a handful of completely arbitrarily designed tests.

We've barely convinced ourselves that LLMs actually increase measurable industry productivity, instead of us just spending time to send slop to each other.

Re: I benchmarked Claude Code's caveman plugin against "be brief."

#48

Post by encody »

"...the value isn't compression. It's structure."

"...that consistency is real value."

"A few findings...are worth flagging here."

I know this smell. I'm not sure if this is AI or merely the natural result of overwhelming immersion in AI output that is "backpropagating" its way into organic communication.

On a completely related note, I've been enjoying classic fiction a lot more recently. Moby Dick is actually pretty funny.

Re: I benchmarked Claude Code's caveman plugin against "be brief."

#50

Post by adrianN »

encody wrote: post #48

"...the value isn't compression. It's structure." "...that consistency is real value." "A few findings...are worth flagging here." I know this smell. I'm not sure if this is AI or merely the natural result of overwhelming immersion in AI output that is "backpropagating" its way into organic communication. On a completely related note, I've been enjoying classic fiction a lot more recently. Moby Dick is actually prett…

A long time ago I slogged through a complete edition of Moby Dick and found it really difficult to read. There are many good passages but they are hidden between endless pages about things that don’t move the plot forward and are only interesting from a sort of historian’s perspective, e.g. treatises about the understanding of whale biology at the time.
Post reply on HN