The first big AI disaster is yet to happen

Post by ahmeni » Mon, Jun 16, 2025, 1:39 AM UTC

soulofmischief wrote: post #28

ahmeni wrote: post #11

The first big AI disaster has already happened and it is the thousand little cuts from people putting faith into systems that aren't built to be trustworthy. It is the software dev missing a critical safety check because "ai coding tools" at work have promised 10x delivery results and now they must use them or be seen as falling behind. It is the engineering manager throwing concrete flexural strength questions into…

This is the research part of research and development, this time though it's sort of like a distributed Manhattan project happening in the open. Research takes a lot of money. Anyone can jump in and help improve these models and the ecosystem.

Post by bwblabs » Mon, Jun 16, 2025, 8:31 PM UTC

TrackerFF wrote: post #59

There's a town in northern Norway where the municipality used AI tools / LLMs to aid their process in restructuring schools. So they were going to close and merge a bunch of schools, and pointed to research that supported their plans. But it turns out that the AI they used, had hallucinated this research. The AI had correctly used relevant researchers, but had hallucinated the work itself. It wasn't until a investiga…