These things are not going to be reliable if you don't know when your session will be routed to inferior model. I stopped using Opus because of that. I had to always create verification task first (a non trivial problem) for the model to prove itself it is "Opus grade" before giving it actual task, but then I found performance often was suddenly severely degraded (model suddenly being dumb as sack of potatoes). This tells me this is not ready for any serious work.
Academic Research Skills for Claude Code
21–30 of 32 posts
Re: Academic Research Skills for Claude Code
#22Academic skills requires proper tools as well, since many research papers are either behind paywalls or protected by bot-detection systems.
Re: Academic Research Skills for Claude Code
#23There needs to be a new name for people creating these with no obvious validation. Skill spam?
The OP evaluates what it has developed with great rigor and describes the evaluation in detail. What do you feel is missing?
It actually does not -- and that is part of the issue. Consumers just see "oh gosh this looks very detailed" and superficially think someone must of spent quite a bit of time on this and it works well.
Skills are just prompts -- and most of what I am seeing are people using AI to write the (quite verbose) prompts. There should be a test, somewhere, that shows "my prompt does better than XYZ other prompt" for some model and some specific inputs. This is what is called a benchmark.
It may work well, I don't know. Just asking Claude "hey help me iterate on a paper" works pretty well out of the box too. Call me skeptical this actually works in any substantive way without seeing any evidence it works.
I agree writing a good benchmark takes time. How do people know if all these prompts they are writing are any good though? You could make an edit and it causes a regression overall. Or add too much info and it is just wasted space in the context window, or causes the model to go in loops between the different skills, or plenty of other errors.
Re: Academic Research Skills for Claude Code
#24There needs to be a new name for people creating these with no obvious validation. Skill spam?
The OP evaluates what it has developed with great rigor and describes the evaluation in detail. What do you feel is missing?
I seriously doubt any human has ever read the full readme for the project.
Re: Academic Research Skills for Claude Code
#25Earlier quoted context omitted.
The OP evaluates what it has developed with great rigor and describes the evaluation in detail. What do you feel is missing?
It actually does not -- and that is part of the issue. Consumers just see "oh gosh this looks very detailed" and superficially think someone must of spent quite a bit of time on this and it works well. Skills are just prompts -- and most of what I am seeing are people using AI to write the (quite verbose) prompts. There should be a test, somewhere, that shows "my prompt does better than XYZ other prompt" for some mod…
I really do run a/b tests. I really do test, and validate.
I do not believe me giving you that information is honest. If I do, I am pretending that you will get the same experience.
Maybe you're using a different model. Maybe you have stuff in your CLAUDE.md that will break it.
It is not honest to me to give you confidence in it, when no one can be confident in it.
Re: Academic Research Skills for Claude Code
#26There needs to be a new name for people creating these with no obvious validation. Skill spam?
Spam. It takes a minute in 2026 to create any app, any skill, any anything without any education that looks plausible that took five years ago a highly educated and skilled person at least months. Now it takes the highly skilled individual ten times the time to evaluate the vibeslopped spam it took the author to publish.
In job I am seeing it as an amplification DoS attack, the amount of content being produced is crippling the processes to protect the org.
Re: Academic Research Skills for Claude Code
#27Earlier quoted context omitted.
The OP evaluates what it has developed with great rigor and describes the evaluation in detail. What do you feel is missing?
It actually does not -- and that is part of the issue. Consumers just see "oh gosh this looks very detailed" and superficially think someone must of spent quite a bit of time on this and it works well. Skills are just prompts -- and most of what I am seeing are people using AI to write the (quite verbose) prompts. There should be a test, somewhere, that shows "my prompt does better than XYZ other prompt" for some mod…
> It actually does not
I read it, right there on the OP. Tests and test results, including discussions of flaws with earlier designs and how they are improved here. What are you talking about?
Re: Academic Research Skills for Claude Code
#28[dead]
Re: Academic Research Skills for Claude Code
#29Earlier quoted context omitted.
Some sort of eval. Eg TermBench, implemented in Harbor. It’s an insane amount of effort to build shareable, reusable, comprehensive evals, hence why so almost all skills are stuck in the “vibes” phase. That said I think it’s quite easy to skim/intuit these sort of skills and do horizontal gene transfer into your own vibes-based system. If you use the skills regularly you can construct a cheap personal eval that is a…
Yeah, I think we're in a phase honestly where you shouldn't use anyone elses skills, and you should instead point your stuff at a repo with skills, have it really read it, and then ask what of value there is to potentially rewrite in your style based on your preferences. I have a complex setup with a lot of things based around what I do. I don't know how anyone could reasonably get their head around any of it. It's a…
Agree, it's impossible to tell if someone else's workflow works with your codebase without actually trying it, which takes time/tokens. I've been thinking about how to make running quick, directional evals easier / more efficient to give more confidence in using / developing skills. Basically, how do we go from vibes to data?