Thanks for the research! Though I feel like industry veterans (especially those working with LLMs) came to this conclusion without having to write a single prompt. Even ignoring the technical merits of these kinds of hacks, if you think you've outwitted billions of dollars of statistics with a prompt, you're probably wrong at this point. What I find most interesting is the popularity of these snake oils, especially t…
Maybe we need a term such as prompt homeopathy to call out prompt engineering without any empirical proof.
Author here. Caveman is a popular Claude Code plugin that compresses Claude's responses via a custom skill with intensity modes. I wanted to know whether it actually beats the simplest possible alternative, prepending "be brief." to prompts. 24 prompts, 5 arms, judged by a separate Claude against per-prompt rubrics covering required facts, required terms, and dangerous wrong claims to avoid. 120 scored responses, 100…
When reading your summary I was wondering how much of those 400 tokens have been consumed by the caveman ruleset.
I tell chats to "be brief" all the time when they're being too verbose, but I never thought to put it in coding agent instructions. Thanks for the benchmark! I wonder how one would put this in AGENTS.md so that it makes sense as a general instruction?
Thanks for the research! Though I feel like industry veterans (especially those working with LLMs) came to this conclusion without having to write a single prompt. Even ignoring the technical merits of these kinds of hacks, if you think you've outwitted billions of dollars of statistics with a prompt, you're probably wrong at this point. What I find most interesting is the popularity of these snake oils, especially t…
The thing is they're not BS when they're released. Prompt Engineering was a real thing that had real results, but then they re-trained the models and now prompt engineering isn't needed on large models. Techniques are gonna vary over time.
Thanks for the research! Though I feel like industry veterans (especially those working with LLMs) came to this conclusion without having to write a single prompt. Even ignoring the technical merits of these kinds of hacks, if you think you've outwitted billions of dollars of statistics with a prompt, you're probably wrong at this point. What I find most interesting is the popularity of these snake oils, especially t…
Maybe we need a term such as prompt homeopathy to call out prompt engineering without any empirical proof.
Thanks for the research! Though I feel like industry veterans (especially those working with LLMs) came to this conclusion without having to write a single prompt. Even ignoring the technical merits of these kinds of hacks, if you think you've outwitted billions of dollars of statistics with a prompt, you're probably wrong at this point. What I find most interesting is the popularity of these snake oils, especially t…
I think there is some benefit to plugins, it's hard to say how much. I find the superpowers plugin is quite good, mostly in its structured approach to a conversation. Generally they do feel pretty overhyped.
Caveman is useless for me. We are in the year 2026, computers are here to serve me, and bring me comfort. Caveman is a caveman, speaks like an idiot. I don't want to interact with an idiot. It's irritating, and as the article states, an overhyped turd. It is the same idiocy that permeates EV cars. You buy an expensive car to go from A to B and at the same time offer you comfort. When I have to think about using the s…
Of the things you could complain about in modern cars as being too complicated, you chose turning on seat heating??? Like you push the seat heating button if your seat feels cold. What is there to think about?
On an electric car that yells at you your range left and that you won't make it to your destination unless you charge, if you turn on the seat warmers, that range goes down so you have to think about if you'd rather have a toasty butt and have to stop and charge, or just be colder and get there sooner. But you have to charge anyway.