Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

95% of generative AI pilots at companies are failing – MIT report

fortune.com

Post reply on HN
Print view View original

51–60 of 174 posts

Re: 95% of generative AI pilots at companies are failing – MIT report

#51

Post by addaon »

svara wrote: post #36
addaon wrote: post #32

Earlier quoted context omitted.

> But I, as a human, rarely have questions to ask. Wow. This just does not match my personal experience. I do an hour or so walk around the reservoir near my house 4-5 times a week, letting my mind wander freely -- and I find that I stop on average at least five or ten times to take notes about questions to learn the answers to later, and occasionally decide that it's worth it to break pace to start learning the answ…

I think not having those instant answers available is a big part of why your mind wanders in that setting.

I have the answers available (I have a phone and good connection), I just am tactical about when to pursue the answer in realtime and when not. If it feels like it's going to open up a wider field of questioning -- or if it feels like I'll learn that this vein is fully mined and goes nowhere -- I'll spend a few minutes searching; otherwise, defer.

Re: 95% of generative AI pilots at companies are failing – MIT report

#53

Post by delusional »

addaon wrote: post #32
agloe_dreams wrote: post #22

Nobody actually wants half the useless tools companies are coming up with because most of the solutions are not really novel. They are just wrapping an LLM. It's kinda like what I realized with the meta Ray-Bans: I can have these things on my face, they can tell me the answer to virtually any question in 10 seconds or less. But I, as a human, rarely have questions to ask. When you walk in to your local grocery store…

> But I, as a human, rarely have questions to ask. Wow. This just does not match my personal experience. I do an hour or so walk around the reservoir near my house 4-5 times a week, letting my mind wander freely -- and I find that I stop on average at least five or ten times to take notes about questions to learn the answers to later, and occasionally decide that it's worth it to break pace to start learning the answ…

I mirror that experience, except for the latter half. I enjoy just being outside and letting my mind wander, letting it wonder about odd questions in the moment. I never actually want or care about the answers, I just like the feeling of thinking.

I already have my phone, I could look up the answers immediately. The reason I don't isn't that I can't. It's that asking the question is the point, not answering it.

Re: 95% of generative AI pilots at companies are failing – MIT report

#54

Post by zero_shift »

Culonavirus wrote: post #48
madcaptenor wrote: post #13

Earlier quoted context omitted.

No. I also thought that even a 95% success rate wouldn't be good enough for airplanes.

It's very much enough for drones tho... all you need is a tiny Jensen's chip, moped engine, some boom boom play-doh and you're ready to rock. No remote control needed.

Drones are expensive. Solid six figures expensive. And they are used around or on things that are even more expensive. You wouldn't want ChatGPT piloting them.

Re: 95% of generative AI pilots at companies are failing – MIT report

#55

Post by addaon »

reactordev wrote: post #37
addaon wrote: post #32

Earlier quoted context omitted.

> But I, as a human, rarely have questions to ask. Wow. This just does not match my personal experience. I do an hour or so walk around the reservoir near my house 4-5 times a week, letting my mind wander freely -- and I find that I stop on average at least five or ten times to take notes about questions to learn the answers to later, and occasionally decide that it's worth it to break pace to start learning the answ…

I rarely have questions of others but I always question myself. :shrug: There’s a difference between asking out loud or another being vs asking yourself internally.

> I rarely have questions of others but I always question myself.

There's only so many questions I have the ability to answer myself. Of those, there's only so many that I have the lifespan to answer myself. We stand on the shoulders of giants, and even on the shoulders of average people -- really it's shoulders all the way down. Unless the questioning itself is the source of joy (which it certainly sometimes is), I prefer to find out what others have learned when they asked the same questions. It's vanishingly rare that I believe I'm the first to think through something.

Re: 95% of generative AI pilots at companies are failing – MIT report

#56

Post by zero_shift »

K0nserv wrote: post #16

I'm arriving at the conclusion that deployments of LLMs is most suitable in areas where the cost of false positives and, crucially, false negatives are low. If you cannot tolerate false negatives I don't see how you get around the inaccuracy of LLMs. As long as you can spot false positives and their rate is sufficiently low they are merely an annoyance. I think this is a good consideration before starting a project l…

I agree, and it's why I think AI is a good $50 billion industry but not a $5 trillion industry.

Re: 95% of generative AI pilots at companies are failing – MIT report

#58

Post by sidewndr46 »

agloe_dreams wrote: post #22

Nobody actually wants half the useless tools companies are coming up with because most of the solutions are not really novel. They are just wrapping an LLM. It's kinda like what I realized with the meta Ray-Bans: I can have these things on my face, they can tell me the answer to virtually any question in 10 seconds or less. But I, as a human, rarely have questions to ask. When you walk in to your local grocery store…

I've tried to express a similar sentiment to people in the past - that 443rd redesign of the UI for JIRA that moves a button from one side to another. It isn't actually for you. You aren't the user of the software. The user of the software is the product manager (or equivalent role). They need to justify their current role or their next promotion.

Re: 95% of generative AI pilots at companies are failing – MIT report

#59

Post by mike_hearn »

aprilthird2021 wrote: post #35
trenchpilgrim wrote: post #6

What's the failure rates if technology pilots in general for comparison? For example, I heard that SAP has an 80-90% deployment failure rate back in the day, but don't have a citable source for it.

That is not remotely true tbh. The company would have failed long ago if it were

Not if every manufacturing company in the world decided to use your software anyway.

ERP rollouts can "fail" for lots of reasons that aren't to do with the software. They are usually business failures. Mostly, companies end up spending so much on trying to endlessly customize it to their idiosyncratic workflows that they exceed their project budgets and abandon the effort. In really bad cases like Birmingham they go live before actually finishing setup, and then lose control of their books and have to resort to hiring people to do the admin manually.

There's a saying about SAP: at some point gaining competitive advantage in manufacturing/retail became all about who could make SAP deployment a success.

This is no different to many other IT projects, most of them fail too. I think people who have never worked in an enterprise context don't realize that; it's not like working in the tech sector. In the tech industry if a project fails, it's probably because it was too ambitious and the tech itself just didn't work well. Or it was a startup whose tech worked, but they couldn't find PMF. But in normal, mature, profitable non-tech businesses a staggering number of business automation projects just fail for social or business reasons.

AI deployments inside companies are going to be like that. The tech works. The business side problems are where the failures are going to happen. Reasons will include:

• Not really knowing what they want the AI to do.

• No way to measure improved productivity, so no way to decide if the API spend is worth it.

• Concluding the only way to get a return is entirely replace people with AI and then having to re-hire them because the AI can't handle the last 5% of the work.

• Non-tech executives doing deals to use models or tech stacks that aren't the right kind or good enough.

etc

Re: 95% of generative AI pilots at companies are failing – MIT report

#60

Post by zero_shift »

onlyrealcuzzo wrote: post #33

These seems like a glass-is-half-empty view. 5% are succeeding. People are trying AI for just about everything right now. 5% is pretty damn good, when AI clearly has a lot of room to get better. The good models are quite expensive and slow. The fast & cheap models aren't that great - unless very specifically fine-tuned. Will it get better enough so that that growth rate in success pilots grows from 5% - 25% in 5 year…

> 5% is pretty damn good, when AI clearly has a lot of room to get better.

That depends if the AI successes depended much on the leading edge of LLM developments, or if actually most of the value was just "low hanging fruit".

If the latter, that would imply the utility curve is levelling out, because new developments are not proving instrumental enough.

I'm thinking of an S curve: slow improvements through the 2010s, then a burst of activity as the tech became good enough to do something "real", followed by more gradual wins in efficiency and accuracy.

Post reply on HN