Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

95% of generative AI pilots at companies are failing – MIT report

fortune.com

Post reply on HN
Print view View original

21–30 of 174 posts

Re: 95% of generative AI pilots at companies are failing – MIT report

#21

Post by nemomarx »

brettgriffin wrote: post #11

> Despite the rush to integrate powerful new models, about 5% of AI pilot programs achieve rapid revenue acceleration; the vast majority stall, delivering little to no measurable impact on P&L. This summer, I built two very sophisticated pieces of software. A financial ledger to power accrual accounting operations and a code generation framework that scaffolds a database from a defined data model to the frontend comp…

I think they mean integrating AI into the business system directly and not using it to code things. I can see that having a more neutral impact

Re: 95% of generative AI pilots at companies are failing – MIT report

#22

Post by agloe_dreams »

Nobody actually wants half the useless tools companies are coming up with because most of the solutions are not really novel. They are just wrapping an LLM.

It's kinda like what I realized with the meta Ray-Bans: I can have these things on my face, they can tell me the answer to virtually any question in 10 seconds or less.

But I, as a human, rarely have questions to ask. When you walk in to your local grocery store - you generally know what you want and where to find it. A ton of companies are just gluing LLM text boxes into apps and then scratching their heads when people don't use them.

Why?

Because the customer wasn't the user - it was their boss and shareholders. It was all done to make someone else think 'woah, they are following the trend!'.

The core issue with generative AI is that it all works best when focused in a narrow sense. There is like one or two really clever uses I've seen - disappointingly, one of them was Jira. The internal jargon dictionary tool was legitimately impressive. Will it make any more money? Probably not.

Re: 95% of generative AI pilots at companies are failing – MIT report

#23

Post by morkalork »

amirkabbara wrote: post #2

Why so bad?

In my experience is that LLMs get you 80%of the way to a solution almost immediately but that last 20% when it comes to missing knowledge, data, or accuracy is a complete tar pit and will wreck adoption. Especially since many vendors are selling products that are wrappers and provide generic, non-customised solutions. I hear the same from others doing trials with various AI tools as well.

Re: 95% of generative AI pilots at companies are failing – MIT report

#25

Post by ModernMech »

brettgriffin wrote: post #11

> Despite the rush to integrate powerful new models, about 5% of AI pilot programs achieve rapid revenue acceleration; the vast majority stall, delivering little to no measurable impact on P&L. This summer, I built two very sophisticated pieces of software. A financial ledger to power accrual accounting operations and a code generation framework that scaffolds a database from a defined data model to the frontend comp…

> But to hear 95% of these efforts don't have a demonstrable effect is just wild.

Why tho? You used AI to make some software, but did you use AI to achieve rapid revenue acceleration?

That you used AI to build software seems tangential to whether it can increase revenues. Over the years, we've seen many technologies that didn't deliver on promises of rapidly increasing revenues despite being useful for creating software (cough OOP cough), so this new one failing to live up to expectations isn't surprising. Actually given the history of technologies that over promise and under deliver on massive hype, disappointment should be the null hypothesis.

Re: 95% of generative AI pilots at companies are failing – MIT report

#26

Post by brettgriffin »

nemomarx wrote: post #21
brettgriffin wrote: post #11

> Despite the rush to integrate powerful new models, about 5% of AI pilot programs achieve rapid revenue acceleration; the vast majority stall, delivering little to no measurable impact on P&L. This summer, I built two very sophisticated pieces of software. A financial ledger to power accrual accounting operations and a code generation framework that scaffolds a database from a defined data model to the frontend comp…

I think they mean integrating AI into the business system directly and not using it to code things. I can see that having a more neutral impact

> Generic tools like ChatGPT excel for individuals because of their flexibility, but they stall in enterprise use since they don’t learn from or adapt to workflows, Challapally explained.

Maybe I misunderstood this, but I took this to mean that people inside enterprises are struggling using tools like ChatGPT. They do point out that perhaps the tools are being deployed in the wrong areas:

> The data also reveals a misalignment in resource allocation. More than half of generative AI budgets are devoted to sales and marketing tools, yet MIT found the biggest ROI in back-office automation—eliminating business process outsourcing, cutting external agency costs, and streamlining operations.

But I've seen some amazing automation does in sales and marketing that directly affected sales efficiency and reduced sales and marketing expenses.

Re: 95% of generative AI pilots at companies are failing – MIT report

#28

Post by thewebguyd »

agloe_dreams wrote: post #22

Nobody actually wants half the useless tools companies are coming up with because most of the solutions are not really novel. They are just wrapping an LLM. It's kinda like what I realized with the meta Ray-Bans: I can have these things on my face, they can tell me the answer to virtually any question in 10 seconds or less. But I, as a human, rarely have questions to ask. When you walk in to your local grocery store…

> There is like one or two really clever uses I've seen - disappointingly, one of them was Jira. The internal jargon dictionary tool was legitimately impressive. Will it make any more money? Probably not.

Sounds like Microsoft 365 Copilot at my org. Sucks at nearly everything, but it actually makes a fantastic search engine for emails, teams convos, sharepoint docs, etc. Much better that Microsoft's own global search stuff. Outside of coding, that's the only other real world use case I've found for LLMs - "get me all the emails, chats, and documents related to this upcoming meeting" and it's pretty good at that.

Though I'm not sure we should be killing the earth for better search, there are probably other, better ways to do it.

Re: 95% of generative AI pilots at companies are failing – MIT report

#29

Post by soiltype »

brettgriffin wrote: post #11

> Despite the rush to integrate powerful new models, about 5% of AI pilot programs achieve rapid revenue acceleration; the vast majority stall, delivering little to no measurable impact on P&L. This summer, I built two very sophisticated pieces of software. A financial ledger to power accrual accounting operations and a code generation framework that scaffolds a database from a defined data model to the frontend comp…

> From the outside, it would appear to any domain expert that at least three other people worked on these giving the pace at which they got completed.

Did several domain experts tell you this or are you making it up?

> I can't imagine how anyone, in any field of information systems, is not multiples more effective than they were five years ago.

Perhaps "they are massively more complex than anything I've done"

Re: 95% of generative AI pilots at companies are failing – MIT report

#30

Post by RaftPeople »

trenchpilgrim wrote: post #6

What's the failure rates if technology pilots in general for comparison? For example, I heard that SAP has an 80-90% deployment failure rate back in the day, but don't have a citable source for it.

> I heard that SAP has an 80-90% deployment failure rate

Something to keep in mind is that ERP "failure" is frequently defined as went over budget or over time, even if it ultimately completed and provided the desired functionality.

It's a much smaller percentage of projects that are either cancelled or went live and significantly did not function as the business needed.

Post reply on HN