Another example of agent skills that give AI agents access to bitdrift's mobile observability platform for full-fidelity agentic investigations -- https://bitdrift.ai/
Agent Skills
141–150 of 239 posts
Re: Agent Skills
#142> A skill is a markdown file with frontmatter that gets injected into the agent’s context when the situation calls for it. When the LLM decides that the situation calls for it > It is a workflow: a sequence of steps the agent follows, with checkpoints that produce evidence, ending in a defined exit criterion. A sequence of steps the LLM can decide to follow
Skills are often invoked imperatively by the user. In cases where they are intended to be used directly by the LLM, it would be included somewhere else in the context. E.g:
``` After implementing the feature, read the testing skill for instructions on how to test. ```
Re: Agent Skills
#143Snake oil. Good to read for sure. Seems all plausible too. But snake oil nevertheless. Here's why: The slot machine can drop any hard requirement that you specifically in your AGENTS.md, memory.md or your dozens of skill markdowns. Pretty much guaranteed. These harnesses approaches pretend as if LLMs are strict and perfect rule followers and the only problem is not being able to specify enough rules clearly enough. T…
Snake oil may be a bit strong, because snake oil never works (except maybe as placebo?) whereas anything with an LLM, even though stochastic, has a pretty high chance of working. > ... you also realize that promised productivity gains are also snake oil because reading code and building a mental model is way harder than having a mental model and writing it into code. Not really, though it depends on the code; reading…
> has a pretty high chance of working.
for MVPs, mock ups, prototypes or in the hands of an expert coder. You can't let them go unsupervised. The promise of automated intelligence falls far short of the reality.
Re: Agent Skills
#144Earlier quoted context omitted.
The primary way to increase reliability is to automate. Instead of humans producing some output manually, humans producing machines which produce that output. I've seen a disturbing trend where a process that could've been a script or a requirement that could've been enforced deterministically is in fact "automated" through a set of instructions for an LLM.
Sure, when that is possible. However, there are lots of processes we don't know how to automate in a deterministic way. Hence the vast amount of investment in building organisations of people with mechanism to make peoples output more reliable through structure, reviews, and so on. Large parts of human civilization rests on our ability to make something unreliable less unreliable through organisational structure and…
We resolve that through liability, penalties, trust, responsibility, review and oversight.
At the end of the day, if I am spending X$s for automation, I want to be able to sleep at night knowing my factory will not build a WMD or delete itself.
If its simply a tool that is a multiplier for experts, then do I really need it? How much does it actually make my processes more efficient, faster, or more capable of earning revenue?
There is a LOT that is forgiven when tech is new - but at some point the shiny newness falls off and it is compared to alternatives.
Re: Agent Skills
#145From an SEO/LLMO perspective, the discoverability of these skills will be difficult without a rename: https://agentskills.io/ If Addy reads this, how do you pitch this vs. Superpowers? https://github.com/obra/superpowers
I would love to know how many people are actually using superpowers. I showed up on the agentic dev scene prior to superpowers, and I am getting concerned that >50% of my self-rolled processes are now covered by superpowers. I no longer trust gh stars, can anyone chime in? Is superpowers now truly adopted? If it is truly valuable, why hasn't Boris integrated the concepts yet?
>I would love to know how many people are actually using superpowers.
I use them on and off. Also Get Shit Done and Compound Engineering. The best results I got with Compound Engineering but it burns tokens like crazy, especially in the review phase where it does reviews with 5 - 12 agents in parallel - and I like to do a lot of reviews for both the plan and documentation and code.
For some lighter tasks, builtin Claude Code skills like plan mode are enough.
Re: Agent Skills
#146I was surprised how long some of these skills are. They are pages and pages long with tables and checkbox lists and code examples, etc. Curious how normal that is - it would only take a couple of these to really fill the context alot.
The reason they are long is because these skills are produced mostly by Claude Code and Opus and no sensible human will read these files, let alone build a mental model around them. There is just layers of assumptions that this works - when in reality it doesn't and it is wasteful. Here is a fun experiment. Ask any LLM to write something vaguely familiar. For example, ask it "write a fib". Since almost all LLMs are f…
I just tried it with Gemini pro. I think this answer is about as good as you can expect for such an ambiguous question.
Write a fib
Since "fib" can mean a couple of different things, I've got you covered for both!
1. A Little Lie (A Fib) "I'm actually typing this to you from a sunny beach in the Bahamas, sipping a piña colada." (Since I'm an AI, that is definitely a fib!)
2. The Fibonacci Sequence If you meant the classic programming exercise, here is a Python function...
Re: Agent Skills
#147Earlier quoted context omitted.
I can take on a slightly weaker form in good faith: professionally it’s a non-starter until private, open source inference can be self-hosted and the ROI is clear enough to invest in that. And on the ROI side, trying things out regularly, I haven’t seen the positive ROI in the limited time I’ve dedicated to exploring the tools. I’ve restricted experimenting to 4 hours per month, because spending more than 2.5% of the…
"I studied math 4 hours per month and I can confirm that mathematics is stupid" You can't learn how to use _anything_ by experimenting 4 hours a month.
The poster provided numbers and thresholds they used to evaluate the utility of a business product.
With infinite time anything is possible, but since we live within constraints, discussing practical, real world thresholds or evaluation methods is a worthwhile use of our time.
Re: Agent Skills
#148Earlier quoted context omitted.
> have been able to successfully track the boost. lets get nitty gritty on this - can you say how you did this? because a lot of people think this is an unsolved problem
For my team, it has been easy. We deal with infrastructure for the entire org, so have tickets created for every request. We also gave our own backlog for internal project, so can see burn rate, and etc. Team hasn’t changed, a lot of similar/same tasks that have taken half a day has been completely automated to a point where we just do PR review after an initial ticket is created by other teams. There are a lot of li…
Thank you for sharing any info at all.
> to be fair, everyone on my team has decade+ professional experience (many more non-prodessional), and we understand limitations of AI fairly well.
I see this appear quite often in discussions on productivity, to the point that a conclusion may be made regarding its centrality for productivity gains.
Re: Agent Skills
#149Earlier quoted context omitted.
Sure, when that is possible. However, there are lots of processes we don't know how to automate in a deterministic way. Hence the vast amount of investment in building organisations of people with mechanism to make peoples output more reliable through structure, reviews, and so on. Large parts of human civilization rests on our ability to make something unreliable less unreliable through organisational structure and…
We resolve that through liability, penalties, trust, responsibility, review and oversight. At the end of the day, if I am spending X$s for automation, I want to be able to sleep at night knowing my factory will not build a WMD or delete itself. If its simply a tool that is a multiplier for experts, then do I really need it? How much does it actually make my processes more efficient, faster, or more capable of earning…
Liability, penalties, trust, and responsibility are means we use to try to influence the application of the processes that do. They do not directly affect reliability. They can be applied just as much to a team using AI as one that does not.
Review and oversight does address reliability directly, and hence why we make use of those in processes to improve the reliability of mechanical processes as well, and why they are core elements of AI harnesses.
> If its simply a tool that is a multiplier for experts, then do I really need it? How much does it actually make my processes more efficient, faster, or more capable of earning revenue?
You can ask the same thing about all the supporting staff around the experts in your team.
> There is a LOT that is forgiven when tech is new - but at some point the shiny newness falls off and it is compared to alternatives.
Only teams without mature processes are not doing that for AI today.
Most of the deployments of AI I work on are the outcome of comparing it to alternatives, and often are part of initiatives to increase reliability of human teams jut as much as increasing raw productivity, because they are often one and the same.
Re: Agent Skills
#150Earlier quoted context omitted.
Humans also drop any hard requirements you specify regularly, and similarly require review. Nevertheless we manage to increase reliability of human output through processes and reviews, and most of the methods we use for harnesses are taken from experience with how to reduce reliability issues in humans, who are notoriously difficult to ensure delivers reliably.
The primary way to increase reliability is to automate. Instead of humans producing some output manually, humans producing machines which produce that output. I've seen a disturbing trend where a process that could've been a script or a requirement that could've been enforced deterministically is in fact "automated" through a set of instructions for an LLM.
it's strange to see software engineers using skills aka human description of small scripts instead of scripting things directly. often there were cli / tools / libraries to do what a skill does for many years. maybe it's culture issue, people who enjoy automation / devops / predictability will naturally help themselves, but other people just want to "delegate" and be done without trying.