actual report here: https://github.com/Papr-ai/papers/blob/main/v0.1%20State%20o...
95% of generative AI pilots at companies are failing – MIT report
141–150 of 174 posts
Re: 95% of generative AI pilots at companies are failing – MIT report
#142Why so bad?
Because for the typical office - documents are strewn about on random network drives and are not formatted similarly. This combined with the inability to nail down 100% accuracy on even just internal doc search is just too much to overcome for non-tech industry offices. My office is mind blown if i use Gemini to extract data from a PDF and convert it to an .xlsx or .csv As a technically minded person but not a comp s…
try https://www.papr.ai for RAG. built it to solve this problem
Re: 95% of generative AI pilots at companies are failing – MIT report
#143The MIT report linked in the article is giving a 404 for some reason. Here is the web archive version: https://web.archive.org/web/20250818145714if_/https://nanda....
Re: 95% of generative AI pilots at companies are failing – MIT report
#144Earlier quoted context omitted.
My walk is also around a reservoir, also 4-5 times a week and the length of the walk around it is also 1 hour. Are you the guy that walks the poodle?
Negative, just myself. I suspect I've mentioned my physical location on HN previously -- southern Utah.
Ah, OK. Wrong state - similar reservoir. There is a guy who walks his poodle at the same time as I walk. We've exchanged head nods for years.
Re: 95% of generative AI pilots at companies are failing – MIT report
#145[deleted]
Re: 95% of generative AI pilots at companies are failing – MIT report
#146Earlier quoted context omitted.
Last year one of my berry bushes had browning leaves with some spots. Google search said infection, treatment plan, etc. This year I snapped a pic and sent to chat gpt. Normal end of year die off, cut the brown branches away, here is a fertilizer schedule for end of year to support new growth for the next year. ChatGPT makes gardening so much easier, and that is just one of many areas. Recipes are another, don't trus…
> This year I snapped a pic and sent to chat gpt. I used to be able to go to the local gardening center and ask the owner who could right away give you the right answer because that was his expertise that came from years of genuine experience. Then Home Depot put him put of business. Same with the local plumbing shop I could walk into with a leaky valve stem from a sink, have a guy glance at it and reply "that's an A…
> I used to be able to go to the local gardening center and ask the owner who could right away give you the right answer because that was his expertise that came from years of genuine experience.
I can still do this, and I do on occasion. Hopefully I take the proper pictures and can remember enough about what is going on to convey the issue. ChatGPT will ask follow up questions and even ask for additional pictures if things aren't clear.
Also I can take action before my once every other month or so visit to the nursery, allowing me to take more immediate action.
Re: 95% of generative AI pilots at companies are failing – MIT report
#147Earlier quoted context omitted.
Has inaccuracies been an issue for any of the systems you have developed using LLMs? I hear your complaint quite a bit but it does not align with my experience. Definitely one shotting a chatbot around an esoteric problem introduces possible inaccuracies. If I get an LLM to interrogate a pdf or other document that error rate drops significantly and is mostly on the part of the structuring process and not the LLM. Gen…
Yes I've seen issues with both, but in part what's tricky about false negatives is also that you don't necessarily realise they are there. In the systems I've worked on we've made it simple for operators to verify the work the LLM has done, but this only guards against false positives, which are less problematic. I've had pretty good success using LLMs for coding and in some ways they are perfect for that. False posi…
> as long as the LLM finds a solution, it's not a huge deal if there was a better way to do it
It might not matter short term, but midterm such debt becomes a huge burden.
Re: 95% of generative AI pilots at companies are failing – MIT report
#148The MIT report linked in the article is giving a 404 for some reason. Here is the web archive version: https://web.archive.org/web/20250818145714if_/https://nanda....
https://github.com/Papr-ai/papers/blob/main/v0.1%20State%20o...
The MIT NANDA lab seems to have a link rot problem.
Their cardinal code repo is also 404. The NANDA Lab also does coding, their publication at AAAI 2025 is titled: "CoDream: Exchanging dreams instead of models for federated aggregation with heterogeneous models" [1]. However, the link to the Github repo is broken. Fascinating paper, sad about the missing code.
Re: 95% of generative AI pilots at companies are failing – MIT report
#149I'm arriving at the conclusion that deployments of LLMs is most suitable in areas where the cost of false positives and, crucially, false negatives are low. If you cannot tolerate false negatives I don't see how you get around the inaccuracy of LLMs. As long as you can spot false positives and their rate is sufficiently low they are merely an annoyance. I think this is a good consideration before starting a project l…
Has inaccuracies been an issue for any of the systems you have developed using LLMs? I hear your complaint quite a bit but it does not align with my experience. Definitely one shotting a chatbot around an esoteric problem introduces possible inaccuracies. If I get an LLM to interrogate a pdf or other document that error rate drops significantly and is mostly on the part of the structuring process and not the LLM. Gen…
I don't really track issues, as I don't need to. Just a recent example "please extract the tabular data from this visual" and the model had incorrect aligned records in one column, so the IDs were off by 1 in the data.
I'm sure in 95% of cases it gets it right, but it didn't this time, and I'm not sure how to actually work around that fact.
Re: 95% of generative AI pilots at companies are failing – MIT report
#150Earlier quoted context omitted.
Why not though? Current autopilot just attempts to keep plane on course/speed/altitude. Some can go further with auto-landing, but extreme emergency use only. I could see the airlines wanting to seek any fuel savings possible by possibly allowing AI to test slight changes to altitude/speed/course to conserve fuel based on some live inputs.
The mathematics that LLMs and machine learning are based on started off being developed for aircraft decades ago. It’s called “control theory”. So we had “AI” on airplanes first. Specifically we had adaptive control algorithms explicitly because of the problems introduced by fuel levels changing during the course of a flight. In physics, we typically start with mass-spring-damper system representation. Elementary phy…
I would say the underlying mathematics is optimization theory, which also has applications in control but was not specifically designed for it (and indeed, the way you approach optimization in control and ML/AI is quite different)