95% of generative AI pilots at companies are failing – MIT report • Hacker News Board

Re: 95% of generative AI pilots at companies are failing – MIT report

#151

Post by red-iron-pine » Tue, Aug 19, 2025, 1:00 PM UTC

MisterTea wrote: post #140

poszlem wrote: post #101

Earlier quoted context omitted.

The origin of the word calf. 1. Calf (young cow, young of certain other mammals) Old English: cealf (plural calfru or later calves) Proto-Germanic: kalbaz or *kalbaz/kalbazō Cognates: Old Norse kálfr, Old High German kalb, German Kalb, Dutch kalf. Proto-Indo-European root: often linked to gel- (“to swell, be rounded”), possibly referring to the rounded shape of a young animal. Some etymologists, however, leave it as…

Literally plugged the phrase "etymology word calf" into duckduckgo and the first result was this: https://etymologyworld.com/item/calf This feels similar to a recent conversation with my friend when I was trying to recall the SoC used in the Nintendo Switch and he insisted on using his chatgpt app when I just went to the Wikipedia page for the Switch faster then he could open his app. I don't want to sound negative,…

eventually they will grow dependent on the tool and will never be able to adapt

Re: 95% of generative AI pilots at companies are failing – MIT report

#152

Post by red-iron-pine » Tue, Aug 19, 2025, 1:02 PM UTC

Culonavirus wrote: post #48

madcaptenor wrote: post #13

Earlier quoted context omitted.

No. I also thought that even a 95% success rate wouldn't be good enough for airplanes.

It's very much enough for drones tho... all you need is a tiny Jensen's chip, moped engine, some boom boom play-doh and you're ready to rock. No remote control needed.

the AI part is simple. getting past GPS jamming and optical dazzlers so that drone AI can function is hard.

Re: 95% of generative AI pilots at companies are failing – MIT report

#153

Post by infecto » Tue, Aug 19, 2025, 1:30 PM UTC

Incipient wrote: post #149

infecto wrote: post #47

Earlier quoted context omitted.

Has inaccuracies been an issue for any of the systems you have developed using LLMs? I hear your complaint quite a bit but it does not align with my experience. Definitely one shotting a chatbot around an esoteric problem introduces possible inaccuracies. If I get an LLM to interrogate a pdf or other document that error rate drops significantly and is mostly on the part of the structuring process and not the LLM. Gen…

I don't really track issues, as I don't need to. Just a recent example "please extract the tabular data from this visual" and the model had incorrect aligned records in one column, so the IDs were off by 1 in the data. I'm sure in 95% of cases it gets it right, but it didn't this time, and I'm not sure how to actually work around that fact.

Not an attack on your experience at all! I would would definitely counter that multimodal are still error prone and much better output is achieved using a tool like textract and then an LLM on the output data.

Re: 95% of generative AI pilots at companies are failing – MIT report

#154

Post by infecto » Tue, Aug 19, 2025, 1:31 PM UTC

K0nserv wrote: post #83

infecto wrote: post #47

Earlier quoted context omitted.

Has inaccuracies been an issue for any of the systems you have developed using LLMs? I hear your complaint quite a bit but it does not align with my experience. Definitely one shotting a chatbot around an esoteric problem introduces possible inaccuracies. If I get an LLM to interrogate a pdf or other document that error rate drops significantly and is mostly on the part of the structuring process and not the LLM. Gen…

Yes I've seen issues with both, but in part what's tricky about false negatives is also that you don't necessarily realise they are there. In the systems I've worked on we've made it simple for operators to verify the work the LLM has done, but this only guards against false positives, which are less problematic. I've had pretty good success using LLMs for coding and in some ways they are perfect for that. False posi…

That’s fair and I typically have utilized LLM workflows where I believe the current gen of models shine. Classifications, data structuring, summarization, etc.

Re: 95% of generative AI pilots at companies are failing – MIT report

#155

Post by ath3nd » Tue, Aug 19, 2025, 1:32 PM UTC

scotty79 wrote: post #65

I remember when it was being said that computers in business had basically the same impact.

Comparing an universal computing machine to what is essentially a fancy autocomplete is just bonkers.

Re: 95% of generative AI pilots at companies are failing – MIT report

#156

Post by HankStallone » Tue, Aug 19, 2025, 1:54 PM UTC

zoeysmithe wrote: post #81

michaelfm1211 wrote: post #20

> The data also reveals a misalignment in resource allocation. More than half of generative AI budgets are devoted to sales and marketing tools, yet MIT found the biggest ROI in back-office automation—eliminating business process outsourcing, cutting external agency costs, and streamlining operations. Makes sense. The people in charge of setting AI initiatives and policies are office people and managers who could be…

I think this is being overly complimenting to AI. I think the most obvious reason is that for almost all business use cases its not very helpful. All these initiatives have the same problem. Staff asking 'how can this actually help me,' because they can't get it to help them other than polishing emails, polishing code, and writing summaries which is not what most people's jobs are. Then you have to proofread all of t…

Part of it is that the bosses often don't know what they want, so they leave the details up to marketing or whoever, so replacing marketing or whoever with AI would mean figuring out what they want. The boss can tell marketing, "Make a brochure for new product ABC," and marketing can run with that and present him with a mock-up, he can make a couple revisions, they shine it up based on those, and then they're done. To replace them completely with AI, he would have to provide a lot more guidance and it would take more iterations to get a correct result that he likes. It wouldn't be completely unlike the current process, but it would demand more of him, which wouldn't make him happy.

Last week I was talking to my boss about a project I've been working on for him, and he asked whether AI could help me with it to save time. I pointed out that a lot of the holdup in the project has been his not knowing exactly what he wants (because he's not sure what the software we're working with can do until I do it and show it to him), and an AI can't tell him what he wants any more than I can. Sometimes you just have to do the work, and technology can't help you.

Re: 95% of generative AI pilots at companies are failing – MIT report

#157

Post by HankStallone » Tue, Aug 19, 2025, 2:05 PM UTC

infecto wrote: post #47

K0nserv wrote: post #16

I'm arriving at the conclusion that deployments of LLMs is most suitable in areas where the cost of false positives and, crucially, false negatives are low. If you cannot tolerate false negatives I don't see how you get around the inaccuracy of LLMs. As long as you can spot false positives and their rate is sufficiently low they are merely an annoyance. I think this is a good consideration before starting a project l…

Has inaccuracies been an issue for any of the systems you have developed using LLMs? I hear your complaint quite a bit but it does not align with my experience. Definitely one shotting a chatbot around an esoteric problem introduces possible inaccuracies. If I get an LLM to interrogate a pdf or other document that error rate drops significantly and is mostly on the part of the structuring process and not the LLM. Gen…

I asked an LLM to guide me through a Salesforce process last week. It gave me step-by-step instructions, about 50% of which were fine while the others referenced options that didn't exist in the system. So I followed the steps until I got to a wrong one, then told it that was wrong, at which point it said that was wrong and gave me different instructions. After a few cycles of that and some trial-and-error, I had a working process.

It probably did save me some time, so I'd call it a mild success; but it didn't save a lot of time, and I only succeeded in the end because I know Salesforce pretty well and was just inexperienced at this one area, so I was able to see where it was probably going off the rails. Someone new to Salesforce would have been hopelessly lost by its advice.

It's understandable that an LLM wouldn't be very good at Salesforce, because there's a lot of bad information in the support forums out there, and the ways of doing things in it have changed multiple times over the years. But that's true of a lot of systems, so it's not an excuse, just a symptom of using LLMs that's probably not going to change.

Re: 95% of generative AI pilots at companies are failing – MIT report

#158

Post by aaronbaugher » Tue, Aug 19, 2025, 2:13 PM UTC

agloe_dreams wrote: post #22

Nobody actually wants half the useless tools companies are coming up with because most of the solutions are not really novel. They are just wrapping an LLM. It's kinda like what I realized with the meta Ray-Bans: I can have these things on my face, they can tell me the answer to virtually any question in 10 seconds or less. But I, as a human, rarely have questions to ask. When you walk in to your local grocery store…

> Because the customer wasn't the user - it was their boss and shareholders.

I'm starting to get asked, "Could AI help you do such-and-such faster?" At first I tried to explain why the answer is no, because such-and-such doesn't lend itself to what AI is good at. But I'm starting to realize I'm going to have to tell them I am using it and maybe give them an example once in a while, because they're hearing too much about its wonderfulness to believe there's something it can't help with. They're going to think I'm just being stubborn even though I tell them I'm not opposed to using AI where it makes sense. If that means the job actually takes a little longer to add in the part where I use AI to speed it up, they'll be happier.

Re: 95% of generative AI pilots at companies are failing – MIT report

#159

Post by zubiaur » Tue, Aug 19, 2025, 3:00 PM UTC

simianwords wrote: post #107

K0nserv wrote: post #16

I'm arriving at the conclusion that deployments of LLMs is most suitable in areas where the cost of false positives and, crucially, false negatives are low. If you cannot tolerate false negatives I don't see how you get around the inaccuracy of LLMs. As long as you can spot false positives and their rate is sufficiently low they are merely an annoyance. I think this is a good consideration before starting a project l…

I completely agree. These are useful in fuzzy cases but we live in a fuzzy world. Most things are fuzzy and nothing is completely true or completely false. If I as a human deploy code, it is not certain that it necessarily works - just like with LLMs. The extent is different however.

100% where we are having a lot of success is in processes that required somewhat repeatable fuzzy processing, which before could only be performed by people.

Cool thing is that, since LLMs are comparatively cheap, I can afford to run the same process a few times, to get a sense of confidence of the response.

In our latest project, the client expressed that our AI aided process was 11 times faster, and much more accurate than their previous process.

Re: 95% of generative AI pilots at companies are failing – MIT report

#160

Post by stockresearcher » Tue, Aug 19, 2025, 5:11 PM UTC

Culonavirus wrote: post #75

zero_shift wrote: post #54

Earlier quoted context omitted.

Drones are expensive. Solid six figures expensive. And they are used around or on things that are even more expensive. You wouldn't want ChatGPT piloting them.

Under $50k for a Geran-2 level drone.

An autonomous Flock drone costs $275k per year:

https://abc7chicago.com/post/oak-brook-police-department-lau...