5% success is actually way higher than I thought it would be. At that rate I suppose there will be actually profitable AI companies with VC subsidies
95% of generative AI pilots at companies are failing – MIT report
41–50 of 174 posts
Re: 95% of generative AI pilots at companies are failing – MIT report
#42Re: 95% of generative AI pilots at companies are failing – MIT report
#43Nobody actually wants half the useless tools companies are coming up with because most of the solutions are not really novel. They are just wrapping an LLM. It's kinda like what I realized with the meta Ray-Bans: I can have these things on my face, they can tell me the answer to virtually any question in 10 seconds or less. But I, as a human, rarely have questions to ask. When you walk in to your local grocery store…
> But I, as a human, rarely have questions to ask. Wow. This just does not match my personal experience. I do an hour or so walk around the reservoir near my house 4-5 times a week, letting my mind wander freely -- and I find that I stop on average at least five or ten times to take notes about questions to learn the answers to later, and occasionally decide that it's worth it to break pace to start learning the answ…
I am in the same boat. I am always thinking about things and recently often asking ChatGPT for an answer. Having a natural language interface for questions has opened the door for me to many more questions.
Re: 95% of generative AI pilots at companies are failing – MIT report
#44Nobody actually wants half the useless tools companies are coming up with because most of the solutions are not really novel. They are just wrapping an LLM. It's kinda like what I realized with the meta Ray-Bans: I can have these things on my face, they can tell me the answer to virtually any question in 10 seconds or less. But I, as a human, rarely have questions to ask. When you walk in to your local grocery store…
> But I, as a human, rarely have questions to ask. Wow. This just does not match my personal experience. I do an hour or so walk around the reservoir near my house 4-5 times a week, letting my mind wander freely -- and I find that I stop on average at least five or ten times to take notes about questions to learn the answers to later, and occasionally decide that it's worth it to break pace to start learning the answ…
Thats super reasonable - I'm a person with ADHD so if I'm asking questions in a grocery store context - I might fully forget things or take way too long to get things done - Going for a walk in nature is absolutely a much better place for questions like that to me though. I think I would prefer to not have tech in the moment to take me out of the space.
Re: 95% of generative AI pilots at companies are failing – MIT report
#45Nobody actually wants half the useless tools companies are coming up with because most of the solutions are not really novel. They are just wrapping an LLM. It's kinda like what I realized with the meta Ray-Bans: I can have these things on my face, they can tell me the answer to virtually any question in 10 seconds or less. But I, as a human, rarely have questions to ask. When you walk in to your local grocery store…
> There is like one or two really clever uses I've seen - disappointingly, one of them was Jira. The internal jargon dictionary tool was legitimately impressive. Will it make any more money? Probably not. Sounds like Microsoft 365 Copilot at my org. Sucks at nearly everything, but it actually makes a fantastic search engine for emails, teams convos, sharepoint docs, etc. Much better that Microsoft's own global search…
Agreed - 95% of the questions I ask Copilot, I could answer myself by searching emails, Teams messages and files - BUT Copilot does a far far better job than me, and quicker. I went from barely using it, to using it daily. I wouldn't say it is a massive speed boost for me, but I'd miss it if it was taken away.
Then the other 5% is the 'extra; it does for me, and gets me details I wouldn't have even known where to find.
But it is just fancy search for me so far - but fancy search I see as valuable.
Re: 95% of generative AI pilots at companies are failing – MIT report
#46Nobody actually wants half the useless tools companies are coming up with because most of the solutions are not really novel. They are just wrapping an LLM. It's kinda like what I realized with the meta Ray-Bans: I can have these things on my face, they can tell me the answer to virtually any question in 10 seconds or less. But I, as a human, rarely have questions to ask. When you walk in to your local grocery store…
> Because the customer wasn't the user - it was their boss and shareholders. It's kinda funny that some online shops are now bragging how great their customer support is because they DON'T use LLM bots xD
Dealing with real humans in the future will be the ultimate VIP treatment.
Re: 95% of generative AI pilots at companies are failing – MIT report
#47I'm arriving at the conclusion that deployments of LLMs is most suitable in areas where the cost of false positives and, crucially, false negatives are low. If you cannot tolerate false negatives I don't see how you get around the inaccuracy of LLMs. As long as you can spot false positives and their rate is sufficiently low they are merely an annoyance. I think this is a good consideration before starting a project l…
Has inaccuracies been an issue for any of the systems you have developed using LLMs? I hear your complaint quite a bit but it does not align with my experience. Definitely one shotting a chatbot around an esoteric problem introduces possible inaccuracies. If I get an LLM to interrogate a pdf or other document that error rate drops significantly and is mostly on the part of the structuring process and not the LLM.
Genuinely curious what others have experienced but specifically those that are using LLMs for business workflows. It is not to say any system is perfect but for purpose driven data pipelines LLMs can be pretty great.
Re: 95% of generative AI pilots at companies are failing – MIT report
#48Am I the only one who looked at this shortened headline and wondered why anyone is allowing AIs to fly airplanes?
No. I also thought that even a 95% success rate wouldn't be good enough for airplanes.
It's very much enough for drones tho... all you need is a tiny Jensen's chip, moped engine, some boom boom play-doh and you're ready to rock. No remote control needed.
Re: 95% of generative AI pilots at companies are failing – MIT report
#49What's the failure rates if technology pilots in general for comparison? For example, I heard that SAP has an 80-90% deployment failure rate back in the day, but don't have a citable source for it.
I think you're on the right track here. Most technology pilots fail. As long as risk/investment is managed appropriately, this is healthy. This seems to follow from Surgeon's Law... 90% of everything is crap [0].
Re: 95% of generative AI pilots at companies are failing – MIT report
#50Am I the only one who looked at this shortened headline and wondered why anyone is allowing AIs to fly airplanes?
Why not though? Current autopilot just attempts to keep plane on course/speed/altitude. Some can go further with auto-landing, but extreme emergency use only. I could see the airlines wanting to seek any fuel savings possible by possibly allowing AI to test slight changes to altitude/speed/course to conserve fuel based on some live inputs.
[deleted]