[deleted]
The current hype around autonomous agents, and what actually works in production
211–220 of 268 posts
Re: The current hype around autonomous agents, and what actually works in production
#212See "Campbell's Completeness Conjecture"
https://www.campbellramble.ai/p/dont-trust-machines
Re: The current hype around autonomous agents, and what actually works in production
#213Earlier quoted context omitted.
Real question: what's the best way to short AI right now?
Just short any of the publicly traded companies with AI based valuations? Nvida, Meta? Seems like an awful idea but I'm often wrong.
Nvidia and Meta are both involved in a lot more than AI. There are maybe other reasons to short Meta, but either is definitely not a pure AI play.
Re: The current hype around autonomous agents, and what actually works in production
#214Very nice article. The point about mathematical reliability is interesting. I generally agree with it, but humans aren't 100% reliable, or even 99% reliable, so how do we manage to create things like the Linux kernel or the Mars landers without AI? Clearly we have some sort of goal-based self-correction mechanism. I wonder if there's research into AI on that thread?
We don't generate chains of tokens with a constant error rate so errors don't pile up. Don't ask me what we do instead for I have no clue but whatever it is, it works better than next token prediction.
Hey, maybe humans aren't just like LLMs after all.
Re: The current hype around autonomous agents, and what actually works in production
#215Earlier quoted context omitted.
You could instruct the LLM to formulate a “lesson” based on the error and add this to the tool instructions for future runs.
This isn’t practical at scale. You’ll run into too many novel lessons and burn through too many tokens setting up context.
At scale you need to use more tricks. For example, only inject examples if the tool is going to be needed. Or amass lessons, then ask the LLM to summarize them to prune redundant information before it is used in the context.
Re: The current hype around autonomous agents, and what actually works in production
#216Human multi-step workflows tend to have checkpoints where the work is validated before proceeding further, as humans generally aren't 99%+ accurate either. I'd imagine future agents will include training to design these checks into any output, validating against the checks before proceeding further. They may even include some minor risk assessment beforehand, such as "this aspect is crucial and needs to be 99% correc…
That's what Claude Code does - it constantly stops and asks you whether you want to proceed, including showing you the suggested changes before they're implemented. Helps with avoiding token waste and 'bad' work.
Except when it decides it doesn’t need to do that anymore or forgets
Re: The current hype around autonomous agents, and what actually works in production
#217Earlier quoted context omitted.
Uptime and reliability are not the same thing. Designing a bridge doesn't require that the engineer be working 99.9% of minutes in a day, but it does require that they be right in 99.9% of the decisions they make.
Another way to think about it is that, if the engineer isn't right in 99.9% of decision, the bridge will have 99.9% uptime.
That's pretty bad for a bridge haha
Re: The current hype around autonomous agents, and what actually works in production
#218Earlier quoted context omitted.
I’ve found claude to be very helpful when both writing and debugging code written in a language i’m currently building. I just make sure to load the spec into its context first and that seems to be enough for it to get a general understanding.
Everyone criticizing AI for not "understanding" anything... yet, as you found, and many others have also shown before, explain something to them and they bloody well look like they do understand it. I am still in awe at what LLMs can do, TBH. Over the last few months, the main problem with them: of confidently making shit up, seems to be getting much less of a problem... it's still not solved, but if things keep impr…
> controls that ensure they stop doing that
I'm not sure you can do that. As humans, we need to make things up in order to have theories to test. Like back in the day before Einstein when people thought that light traveled through an "aether" whose properties we needed to figure out how to measure, or today when we can't explain the mass imbalance of the universe so we create this concept called "dark matter."
Also, in my experience the problem has been getting worse, or at least not better. I asked Claude 3.7 some time ago how to restore a snapshot to an active database on AWS, and it cheerfully told me to go to the console and press the button. Except there is no button, because AWS docs specifically say you can't restore a snapshot to an active database.
Re: The current hype around autonomous agents, and what actually works in production
#219Very nice article. The point about mathematical reliability is interesting. I generally agree with it, but humans aren't 100% reliable, or even 99% reliable, so how do we manage to create things like the Linux kernel or the Mars landers without AI? Clearly we have some sort of goal-based self-correction mechanism. I wonder if there's research into AI on that thread?
I believe there was an article/paper in the last few months about that exact issue Someone was saying that with an increasing number of attempts, or increasing context length, LLMs are less and less likely to solve a problem (I searched for it but can't find it) That matches my experience -- the corrections in long context can just as easily be anti-corrections, e.g. turning something that works into something that d…
I saw your edit with the paper, but when you first mentioned it I thought you might have been referring to the Apple paper that more or less said the same thing.
Speaking of Apple, I just want to get it out there that I'm impressed that they're exhibiting self restraint in this AI era. I know they get bashed for not being "up to speed" with "the rest of the industry," but I believe they're doing this on purpose because they see what slop it is and they'd prefer to scope it down to release something more useful.
Re: The current hype around autonomous agents, and what actually works in production
#220I spoke with an Amazon AI production engineer who’s talking with prospective clients about implementing AI in our business. When a colleague asked about using generative AI in customer facing chats the engineer said he knows of zero companies who don’t have a human in the loop. All the automatic replies are non-generative “old” tech. Gen AI is just not reliable enough for anyone to stake their reputation on it.
Plenty of tech companies have started using gen AI for live chat support. Off the top of head I know off sonder.com and wealthsimple.com. If the LLM can’t answer a query it usually forwards the chat to a human support agent.
Air Canada did this a bit ago, and their AI gave the customer a fake process for submitting claims for some sort of discount on airfare due to bereavement (the flight was for a funeral). The customer sued and Air Canada's defense was that he shouldn't have trusted the Air Canada AI chatbot. Air Canada lost.