21–30 of 60 posts
AI outputs often feel like a gacha game. Paradoxically, the 'expensive' tokens are sometimes the cheapest in the long run. In my experience, higher-end models have a much higher 'one-shot' success rate. You aren't just saving on total token count by avoiding loops; you’re saving engineering time, which is always the most expensive resource anyway.
Earlier quoted context omitted.
yes, compaction and smaller models help on cost per step. But my issue wasn’t just inefficiency, it was agents retrying when they shouldn’t. I needed visibility + limits per agent/task, and the ability to cut it off, not just optimize it.
I'm working on a fun project I call OpenFAST, which essentially tries to solve the context transitioning - but its still in early days and haven't released anything yet. I think one of the bigger issues, is the o(n) orchestration to agent calls that often feels uncontrolled .. ending up making the orchestrator of sub-agents the main bottleneck due to the large context it sometimes ends up with. I'm working on an idea…
Here's something which would help you with another perspective on the contexts https://authority.bhaviavelayudhan.com/journal/35
cost control is a policy problem - we certainly don't need to use opus 4.6 for a simple test refactor, but many people (including myself) default to it anyways. we need a way to measure cost / performance for agents on individual repos, with individual types of tasks, to get a better sense of what tasks can be trusted to cheaper agents, and what tasks must be routed to the SOTA
But cost control is not an entirely policy problem. Policies are just guidelines.
Don't use tech with deep, unresolved flaws and you won't get fucked. Would you find it acceptable if Postgresql occassionally hallucinated and returned gibberish? Fuck no. Wny is this okay with ANY software? Answer, it's not. AI IS NOT READY.
[flagged]
Mine have burned a lot of money! Right now, I'm trying to keep the context smaller. It takes a lot of discipline, though, to have a system that gives enough context to do the work but not so much the agent can go off doing new/crazy stuff.