[stub for offtopicness]
Don't tell management about this, as they're all betting the house on AI agents next year.
The current hype around autonomous agents, and what actually works in production
21–30 of 268 posts
Re: The current hype around autonomous agents, and what actually works in production
#22Re: The current hype around autonomous agents, and what actually works in production
#23> Production systems need 99.9%+ reliability This is not remotely true. Think of any business process around your company. 99.9% availability would mean only 1min26 per day allowed for instability/errors/downtime. Surely your human collaborators aren't hitting this SLA. A single coffee break immediately breaks this (per collaborator!). Business Process Automation via AI doesn't need to be perfect. It simply needs to…
Currently I'm thinking about how furious the developers get any time Jenkins has any kind of hiccough, even if the solution is just "re-run the workflow" - and that's just network timeouts! I don't want to imagine the tickets if the CI system started spitting out hallucinations...
Re: The current hype around autonomous agents, and what actually works in production
#24> Each new interaction requires processing ALL previous context I was under the impression that some kind of caching mechanism existed to mitigate this
My understanding is that caching reduce computation but the whole input is still processed. I don’t think is fully disclosing how their cache works.
LLMs degrade with long input regardless of caching.
Re: The current hype around autonomous agents, and what actually works in production
#25Human multi-step workflows tend to have checkpoints where the work is validated before proceeding further, as humans generally aren't 99%+ accurate either. I'd imagine future agents will include training to design these checks into any output, validating against the checks before proceeding further. They may even include some minor risk assessment beforehand, such as "this aspect is crucial and needs to be 99% correc…
Lots of applications have to be redesigned around that. My guess is that micro-services architecture will see a renaissance since it plays well with LLMs.
Re: The current hype around autonomous agents, and what actually works in production
#26> Production systems need 99.9%+ reliability This is not remotely true. Think of any business process around your company. 99.9% availability would mean only 1min26 per day allowed for instability/errors/downtime. Surely your human collaborators aren't hitting this SLA. A single coffee break immediately breaks this (per collaborator!). Business Process Automation via AI doesn't need to be perfect. It simply needs to…
I think you're crossing reliability and availability.
Reliability means 99.9% of the time when I hand something off to someone else it's what they want.
Availability means I'm at my desk and not at the coffee machine.
Humans very much are 99.9% accurate, and my deliverable even comes with a list of things I'm not confident about
Re: The current hype around autonomous agents, and what actually works in production
#27> Each new interaction requires processing ALL previous context I was under the impression that some kind of caching mechanism existed to mitigate this
You have to compute attention between all pairs of tokens at each step, making the naive implementation O(N^3). This is optimized by caching the previous attention values, so that for each step you only need to compute attention between the new token and all previous ones. That's much better but still O(N^2) to generate a sequence of N tokens.
Re: The current hype around autonomous agents, and what actually works in production
#28Earlier quoted context omitted.
Caching would only help to keep the context around, but caching would only be needed if it still ultimately needs to read and process that cached context again.
You can cache the whole inference state, no? They don't go into implementation details but Gemini docs say you get a 75% discount if there's a context-cache hit: https://cloud.google.com/vertex-ai/generative-ai/docs/contex...
It that just avoids having to send the full context for follow-up requests, right? My understanding is that caching helps to keep the context around but can't avoid the need to process that context over and over during inference.
Re: The current hype around autonomous agents, and what actually works in production
#29[stub for offtopicness]
"Let's do the math. "
This phrase is usually followed by some, you know...Math?
Re: The current hype around autonomous agents, and what actually works in production
#30> Production systems need 99.9%+ reliability This is not remotely true. Think of any business process around your company. 99.9% availability would mean only 1min26 per day allowed for instability/errors/downtime. Surely your human collaborators aren't hitting this SLA. A single coffee break immediately breaks this (per collaborator!). Business Process Automation via AI doesn't need to be perfect. It simply needs to…
It's not just about up time. If the bridge collapses people die. Some of us aren't selling ads.