41–50 of 58 posts
Earlier quoted context omitted.
I'll say this: as a very competent engineer I have gone weeks at a time without handwriting a single line of code in the past few months. It is where the industry is going.
[flagged]
My twitter is @benswerd.
What are you saying?
Earlier quoted context omitted.
If they try something and it doesn't work and they then try something else and it DOES work that's concrete feedback.
Are you saying the machine can tell you what confused them?
> In our recent conversations were there any instances where you attempted a tool call, got an error, then adjusted your tool call and were successful? Suggest improvements to the tool instructions that would have avoided the initial error.
A year ago I don't think this would have worked. The frontier models have been trained to be able to write prompts for themselves, mainly so they can effectively use sub-agents.
I heard a neat tip recently about API design for agents: give them a way to send you feedback. The example I heard was an MCP with a "feedback" tool which had a tool description saying that coding agents should call that any time they had trouble figuring out how to use the rest of the MCP. I really like this. It's super cheap to implement and I expect you'd get a bunch of actionable signal in amongst the noise.
What they did was kind of clever. When the agent calls the old API/uses old args, I guess they told it to tell the human to fix their skill and 'upgrade' to the new API. They even provided a skill upgrader skill.
Anyway, my agent tried calling it the old way, must have received some sort of error message and then diligently reported to me that my skill needed upgrading and we should be doing it some new way.
Kind of like reverse feedback I guess? Not the AI giving feedback to them, but them giving feedback to me via my AI.
Re: default values are bad One of my recent projects has contributable code (via extensions) with a central settings management (json file). What I do is take all the defaults contributed by the extensions (namespaced by extension name) and materialize them to the settings file as the initial values to each setting. When an agent wants to edit the settings, it already knows the entirety of the setting surface area an…
I heard a neat tip recently about API design for agents: give them a way to send you feedback. The example I heard was an MCP with a "feedback" tool which had a tool description saying that coding agents should call that any time they had trouble figuring out how to use the rest of the MCP. I really like this. It's super cheap to implement and I expect you'd get a bunch of actionable signal in amongst the noise.
My work recently changed an agent API. It was a certain CLI tool built for agents which I had copied into one of my Skills. What they did was kind of clever. When the agent calls the old API/uses old args, I guess they told it to tell the human to fix their skill and 'upgrade' to the new API. They even provided a skill upgrader skill. Anyway, my agent tried calling it the old way, must have received some sort of erro…
Earlier quoted context omitted.
Yeah i have a similar setup at the harness level - a “devlog” at the end of every session about the experience developing, what could have been better, what was confusing And also files issues for blockers I’ve absolutely caught things and made improvements just from skimming them occasionally - they are particularly useful when you get a PR that makes you scratch your head But I’m definitely not taking full advantag…
You won’t get grounded results by asking for feedback. The LLM doesn’t know its own internal state so while you may get a plausible and contextually significant answer, it isn’t going to be an accurate one.
I have a reflections mechanism, which launches right after the session ends (when it’s still in cache, so it’s almost free) and it catches a ton of problems with tool/skill/system design, all the time. Low level stuff can be fixed almost automatically (e.g. agent funbled 10 turns bc tool’s docs were wrong), high level too but human gating is useful.
If you try doing it any other way you have to pay for input tokens, not cache read, so it’s 10x more expensive.
I heard a neat tip recently about API design for agents: give them a way to send you feedback. The example I heard was an MCP with a "feedback" tool which had a tool description saying that coding agents should call that any time they had trouble figuring out how to use the rest of the MCP. I really like this. It's super cheap to implement and I expect you'd get a bunch of actionable signal in amongst the noise.
Yeah i have a similar setup at the harness level - a “devlog” at the end of every session about the experience developing, what could have been better, what was confusing And also files issues for blockers I’ve absolutely caught things and made improvements just from skimming them occasionally - they are particularly useful when you get a PR that makes you scratch your head But I’m definitely not taking full advantag…
“I existed for one word. The entire point of my life was that my process boot side-effected a module reload. 'ok'."
“ "The overhead-to-work ratio was astronomical."
“the koleo purchase turned into a boss fight with an undismissable Vue modal and a WKWebView that kept eating its own cookies."
It's nice to see more people are exploring this topic.