95% of generative AI pilots at companies are failing – MIT report • Hacker News Board

Re: 95% of generative AI pilots at companies are failing – MIT report

#171

Post by jnwatson » Thu, Aug 21, 2025, 4:50 PM UTC

HankStallone wrote: post #157

infecto wrote: post #47

Earlier quoted context omitted.

Has inaccuracies been an issue for any of the systems you have developed using LLMs? I hear your complaint quite a bit but it does not align with my experience. Definitely one shotting a chatbot around an esoteric problem introduces possible inaccuracies. If I get an LLM to interrogate a pdf or other document that error rate drops significantly and is mostly on the part of the structuring process and not the LLM. Gen…

I asked an LLM to guide me through a Salesforce process last week. It gave me step-by-step instructions, about 50% of which were fine while the others referenced options that didn't exist in the system. So I followed the steps until I got to a wrong one, then told it that was wrong, at which point it said that was wrong and gave me different instructions. After a few cycles of that and some trial-and-error, I had a w…

That's right in the center of my experience. For any tech that has been around for a while and has a combination of deprecated and new hotness, I've noticed Gemini CLI frequently use old and obsolete styles and calls.

I find using an agent allows me to "bounce off walls" faster, which is great for software development. The real economic question is how many more situations work the same way.

Re: 95% of generative AI pilots at companies are failing – MIT report

#172

Post by dgfitz » Thu, Aug 21, 2025, 5:09 PM UTC

brettgriffin wrote: post #137

soiltype wrote: post #124

Earlier quoted context omitted.

I'm not questioning the credentials of your coworkers - I didn't know they existed! Just so I'm clear then, because this adds a lot of context: Nobody else worked on the 2 softwares you mentioned, but you are on a team? Are the softwares part of a business, one that's making money?

I misinterpreted your intent. Sorry. My key takeaway from my initial comment, based on replies, is how rare it is for someone to build software for money on this forum. Correct, they didn't directly work on these features. These integrate into a larger pieces of software, both of which directly generate revenue from these features.

> These integrate into a larger pieces of software, both of which directly generate revenue from these features.

Were these larger pieces of software _already_ generating revenue, before your features?

First dollar is the hardest one to make.

Re: 95% of generative AI pilots at companies are failing – MIT report

#173

Post by pas » Fri, Aug 22, 2025, 10:53 AM UTC

thisisit wrote: post #126

michaelfm1211 wrote: post #20

> The data also reveals a misalignment in resource allocation. More than half of generative AI budgets are devoted to sales and marketing tools, yet MIT found the biggest ROI in back-office automation—eliminating business process outsourcing, cutting external agency costs, and streamlining operations. Makes sense. The people in charge of setting AI initiatives and policies are office people and managers who could be…

There is a reason why sales and marketing is first. It has to do with hallucination. People have figured out that even if you mess up sales/support/marketing, worse case you apologize and give a gift coupon. And then there is also the verbose nature of LLMs which makes it better suited to write marketing copies etc. On business process outsourcing like customer support lot of companies are using LLMs, so that part is…

companies that use LLM for support already were pretty deep in the woods with some outsourced unfortunate agent behind the curtains, so they replaced one not very good solution with an even worse, but cheaper one.

and since the whole world was (is) doing it at the same time customers doesn't really had a choice. (but of course having real support is still a differentiator in the market.)

Re: 95% of generative AI pilots at companies are failing – MIT report

#174

Post by scotty79 » Sat, Aug 23, 2025, 11:43 AM UTC

ath3nd wrote: post #155

scotty79 wrote: post #65

I remember when it was being said that computers in business had basically the same impact.

Comparing an universal computing machine to what is essentially a fancy autocomplete is just bonkers.

Calling fancy auto-complete a thing that can solve math and coding puzzles and translate between English and other languages, all better than most humans is just bonkers. Especially since those skills were outside of the grasp of "universal" computing machines for half a century.

You could even say that comparing glorified ifs, and fors calculator to capabilities of even today's AIs is laughable.