Live data from Hacker News

95% of generative AI pilots at companies are failing – MIT report

fortune.com

131–140 of 174 posts

Re: 95% of generative AI pilots at companies are failing – MIT report

#131

Earlier quoted context omitted.

I was going to say the same. It's probably so much healthier to make note of questions for later research than to stop right then and there and either a) fall down a Wikipedia rabbit hole or b) have an AI strapped to your face perform an info-dump.

Not everyone wants an imagination. This is good for those who don't.

[deleted]

Re: 95% of generative AI pilots at companies are failing – MIT report

#132
We've talked with a ton of AI companies and I was surprised how much of the challenges were the usual challenges in any project. Just amplified by the rush to do AI right now, but I haven't seen anything as bad as "You couldn’t have customer calls, you couldn’t work on budgets, you had to only work on AI projects.”

Warning for gratuitous self promotion: https://humansignal.com/blog/9-criteria-for-successful-ai-pr...

Re: 95% of generative AI pilots at companies are failing – MIT report

#133
post #82

Nobody actually wants half the useless tools companies are coming up with because most of the solutions are not really novel. They are just wrapping an LLM. It's kinda like what I realized with the meta Ray-Bans: I can have these things on my face, they can tell me the answer to virtually any question in 10 seconds or less. But I, as a human, rarely have questions to ask. When you walk in to your local grocery store…

I think those kind of glasses may be really useful for blind people. I have seen similar glasses targeted at blind people, that at least in theory, seemed to me like a good idea. I recall the glasses also can write on the screen inside the lens, which makes me think they may be good for deaf people as well. It's just that these use-cases seem uncool, and big companies seem to have to be cool in order to keep either t…

Oh I do still enjoy the glasses, they are actually rather incredible, even though they do not have a screen. That said - These actually do have a Be My Eyes integration - It is incredibly impressive.

Re: 95% of generative AI pilots at companies are failing – MIT report

#134
post #16

I'm arriving at the conclusion that deployments of LLMs is most suitable in areas where the cost of false positives and, crucially, false negatives are low. If you cannot tolerate false negatives I don't see how you get around the inaccuracy of LLMs. As long as you can spot false positives and their rate is sufficiently low they are merely an annoyance. I think this is a good consideration before starting a project l…

I'm working on some AI projects and I'm building in "what just happened" kinda interface so folks understand if the result is in fact is what they wanted.

Management types seem baffled by the idea we would want this, even if they come around the next hour and say "hey user did something can you tell me what happened".

Like guies ... it's not 100%...

Re: 95% of generative AI pilots at companies are failing – MIT report

#135

Earlier quoted context omitted.

> I, as a human, rarely have questions to ask This is an eye-opening sentence. It's quite hard to imagine how to live one's daily life with "few questions to ask." Perhaps this is a neurodivergent thing?

I always ponder how many people have a refrigerator in their home their entire life, and what percentage of them don't know how it works. I've asked several gfs, and they don't have even a hint of how it works. Guy friends do a bit better but not as well as you'd think. So yes, people live their entire lives not asking obvious questions.

some of us have other things to do

Re: 95% of generative AI pilots at companies are failing – MIT report

#136

Nobody actually wants half the useless tools companies are coming up with because most of the solutions are not really novel. They are just wrapping an LLM. It's kinda like what I realized with the meta Ray-Bans: I can have these things on my face, they can tell me the answer to virtually any question in 10 seconds or less. But I, as a human, rarely have questions to ask. When you walk in to your local grocery store…

> I, as a human, rarely have questions to ask This is an eye-opening sentence. It's quite hard to imagine how to live one's daily life with "few questions to ask." Perhaps this is a neurodivergent thing?

I meant mostly in the context of daily life tasks as a person with ADHD - so maybe a hair neurodivergent. My issue isn't that I don't wonder things, it is that indulging the wonder would interrupt me from accomplishing almost anything. I would not very highly functioning if I allowed for non-critical thoughts to interrupt the flow. When outside of trying to do specific things and in less focus-dependent tasks, I absolutely wonder and google and get lost on weird random topics.

I think I probably could have worded it more as "I rarely have questions worth knowing the answer to", where the cost of knowing answers is tied to the following rabbit holes and delays/forgotten tasks.

Re: 95% of generative AI pilots at companies are failing – MIT report

#137

Earlier quoted context omitted.

> Did several domain experts tell you this or are you making it up? It's an assertion among eight other engineers on the project with ~15 years of experience in the domain. They are domain experts. This part isn't up for debate.

I'm not questioning the credentials of your coworkers - I didn't know they existed! Just so I'm clear then, because this adds a lot of context: Nobody else worked on the 2 softwares you mentioned, but you are on a team? Are the softwares part of a business, one that's making money?

I misinterpreted your intent. Sorry. My key takeaway from my initial comment, based on replies, is how rare it is for someone to build software for money on this forum.

Correct, they didn't directly work on these features. These integrate into a larger pieces of software, both of which directly generate revenue from these features.

Re: 95% of generative AI pilots at companies are failing – MIT report

#138
post #84

> Despite the rush to integrate powerful new models, about 5% of AI pilot programs achieve rapid revenue acceleration; the vast majority stall, delivering little to no measurable impact on P&L. This summer, I built two very sophisticated pieces of software. A financial ledger to power accrual accounting operations and a code generation framework that scaffolds a database from a defined data model to the frontend comp…

“AI pilots” in the article refers to developing AI-based tools, not to using AI for software development. These projects have a 95% failure rate of successfully deploying the AI tool being developed into production. Regarding use of AI in software development (which is not what the article is about), the proof of the pudding isn’t in greenfield projects, it’s in longer-term software evolution and legacy code. Few dis…

You are correct. As I pointed out in another reply, I misinterpreted this part:

> Generic tools like ChatGPT excel for individuals because of their flexibility, but they stall in enterprise use since they don’t learn from or adapt to workflows, Challapally explained.

I didn't read the actual report (and probably wont) so I was figured "AI Pilots" _could_ (and honestly, should!) include the deployment of models to assist in any and all work (not necessarily even just coding - I just used it as an example).

Re: 95% of generative AI pilots at companies are failing – MIT report

#139
post #98

Earlier quoted context omitted.

But do you need AI for those answers? I sometimes do the same thing, but Google/DDG/whatever works fine for most, and a niche app works for others (IDing a bird = Merlin app, for example).

Last year one of my berry bushes had browning leaves with some spots. Google search said infection, treatment plan, etc. This year I snapped a pic and sent to chat gpt. Normal end of year die off, cut the brown branches away, here is a fertilizer schedule for end of year to support new growth for the next year. ChatGPT makes gardening so much easier, and that is just one of many areas. Recipes are another, don't trus…

> This year I snapped a pic and sent to chat gpt.

I used to be able to go to the local gardening center and ask the owner who could right away give you the right answer because that was his expertise that came from years of genuine experience. Then Home Depot put him put of business. Same with the local plumbing shop I could walk into with a leaky valve stem from a sink, have a guy glance at it and reply "that's an American Standard" spin around, open a drawer and hand me the part along with new washers.

Now I have to talk to a computer that may or may not be correct. I would rather talk to a real person.

Re: 95% of generative AI pilots at companies are failing – MIT report

#140

Earlier quoted context omitted.

What was the last thing you asked about? What was the answer?

The origin of the word calf. 1. Calf (young cow, young of certain other mammals) Old English: cealf (plural calfru or later calves) Proto-Germanic: kalbaz or *kalbaz/kalbazō Cognates: Old Norse kálfr, Old High German kalb, German Kalb, Dutch kalf. Proto-Indo-European root: often linked to gel- (“to swell, be rounded”), possibly referring to the rounded shape of a young animal. Some etymologists, however, leave it as…

Literally plugged the phrase "etymology word calf" into duckduckgo and the first result was this: https://etymologyworld.com/item/calf

This feels similar to a recent conversation with my friend when I was trying to recall the SoC used in the Nintendo Switch and he insisted on using his chatgpt app when I just went to the Wikipedia page for the Switch faster then he could open his app.

I don't want to sound negative, but - to me people who over rely on LLMs are lazy and low effort. I would not hire or work with them.

Post reply on HN