Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Armed police swarm student after AI mistakes bag of Doritos for a weapon

dexerto.com

Post reply on HN
Print view View original

221–230 of 468 posts

Re: Armed police swarm student after AI mistakes bag of Doritos for a weapon

#221

Post by b00ty4breakfast »

neilv wrote: post #91

> Omnilert later admitted the incident was a “false positive” but claimed the system “functioned as intended,” saying its purpose is to “prioritize safety and awareness through rapid human verification.” It just created a situation in which a bunch of people with guns were told that some teen had a gun. That's a very unsafe situation that the system created, out of nothing. And some teen may be traumatized. Again, un…

>...its purpose is to “prioritize safety and awareness through rapid human verification.

Oh look, a corporation refusing to take responsibility for literally anything. How passe.

Re: Armed police swarm student after AI mistakes bag of Doritos for a weapon

#222

Post by trehalose »

janalsncm wrote: post #148
neilv wrote: post #91

> Omnilert later admitted the incident was a “false positive” but claimed the system “functioned as intended,” saying its purpose is to “prioritize safety and awareness through rapid human verification.” It just created a situation in which a bunch of people with guns were told that some teen had a gun. That's a very unsafe situation that the system created, out of nothing. And some teen may be traumatized. Again, un…

In any system, there are false positives and false negatives. In some situations (like a high recall disease detection) false negatives are much worse than false positives, because the cost of a false positive is a more rigorous screening. But in this case both are bad. If it was a false negative students might need therapy for a more tragic reason. Aside from improving the quality of the detection model, we should t…

False positives can effectively lead to false negatives too. If too many alarms end in teens getting swatted (or worse) for eating chips, people might ignore the alarm if an actual school shooter triggers it. Might assume the AI is just screaming about a bag of chips again.

Re: Armed police swarm student after AI mistakes bag of Doritos for a weapon

#223

Post by underlipton »

MisterTea wrote: post #199
adaml_623 wrote: post #195

Earlier quoted context omitted.

This is a really good question. Sadly the answer is that they think it's how the system is meant to work. Well that seems to be the answer that I see coming from police spokespeople

Its likely procedure that they have to follow (see my other post in this thread.) I hate to say this but I get it. Imagine a scenario happens where they decide "sounds phony. stand down." only for it to be real and people are hurt/killed because the "cops ignored our pleas for help and did nothing." which would be a horrible mistake they could be liable for, never mind the media circus and PR damage. So they treat al…

Cops don't have a duty to protect people, so "cops ignored our pleas for help and did nothing" is a-ok, no liability (thank you, qualified immunity). They very much do not treat all scenarios as real; they go gung-ho when they want to and hang back for a few hours "assessing the situation" when they don't.

Re: Armed police swarm student after AI mistakes bag of Doritos for a weapon

#224

Post by michaelt »

omnipresent12 wrote: post #101
neilv wrote: post #91

> Omnilert later admitted the incident was a “false positive” but claimed the system “functioned as intended,” saying its purpose is to “prioritize safety and awareness through rapid human verification.” It just created a situation in which a bunch of people with guns were told that some teen had a gun. That's a very unsafe situation that the system created, out of nothing. And some teen may be traumatized. Again, un…

https://www2.ljworld.com/news/schools/2025/aug/07/lawrence-s... Another false positive by one of these leading content filters schools use - the kid said something stupid in a group chat and an AI reported it to the school, and the school contacted the police. The kid was arrested, stripped searched, and held for 24 hours without access to their parents or counsel. They ultimately had to spend time in probation, a fu…

> They are suing Gaggle, who claims they never intended their system to be used that way.

Yeah, there's a shop near me that sells bongs "intended" for use with tobacco only.

Re: Armed police swarm student after AI mistakes bag of Doritos for a weapon

#225

Post by nomel »

xbar wrote: post #208
tartoran wrote: post #88

"“We understand how upsetting this was for the individual that was searched as well as the other students who witnessed the incident,” the principal wrote. “Our counselors will provide direct support to the students who were involved.”" Make them pay money for false positives instead of direct support and counselling. This technology is not ready for production, it should be in a lab not in public buildings such as s…

Charge the superintendent with swatting. Decision-maker accountability is the only thing that halts bad decision-making.

> Charge the superintendent with swatting.

This assumes no human verification of the flagged video. Maybe the bag DID look like a gun. We'll never know, because modern journalism has no interest in such things. They obtained the required emotional quotes and moved on.

Re: Armed police swarm student after AI mistakes bag of Doritos for a weapon

#226

Post by Lammy »

nkrisc wrote: post #181
janalsncm wrote: post #148

Earlier quoted context omitted.

In any system, there are false positives and false negatives. In some situations (like a high recall disease detection) false negatives are much worse than false positives, because the cost of a false positive is a more rigorous screening. But in this case both are bad. If it was a false negative students might need therapy for a more tragic reason. Aside from improving the quality of the detection model, we should t…

In this case false positives are far, far worse than false negatives. A false negative in this system does not mean a tragedy will occur, because there are many other preventative measures in place. And never mind the fact that this country refuses to even address the primary cause of gun violence in the first place: the ubiquity of guns in our society. So systems like this is what we end up with when we ignore to ad…

> the primary cause of gun violence in the first place: the ubiquity of guns in our society

I would have gone with “a normalized sense of hopelessness and indignity which causes people to feel like violence is the only way they can have any agency in life” considering “gun” is the adjective and “violence” is the actual thing you're talking about.

Re: Armed police swarm student after AI mistakes bag of Doritos for a weapon

#227

Post by wartywhoa23 »

MiiMe19 wrote: post #77
mentalgear wrote: post #18

Ah, the coming age of Palantir's all seeing platform; and Peter Thiel becoming the shadow Emperor. Too bad non-deterministic ml systems are prone to errors that risk lives when applied wrongly to crucial parts of society. But in an authoritarian state, those will be hidden away anyway, so there's nothing to see here: move along folks. Yes, surveillance and authoritarianism go hand in hand, ask China. It's important t…

I might be missing something but I don' think this article isn't about palantir or any of their products

Palantir is but one head of the hydra which has hundreds of them, and all concerns about a single one apply to the whole beast hundredfold.

Re: Armed police swarm student after AI mistakes bag of Doritos for a weapon

#228

Post by DrewADesign »

neilv wrote: post #91

> Omnilert later admitted the incident was a “false positive” but claimed the system “functioned as intended,” saying its purpose is to “prioritize safety and awareness through rapid human verification.” It just created a situation in which a bunch of people with guns were told that some teen had a gun. That's a very unsafe situation that the system created, out of nothing. And some teen may be traumatized. Again, un…

Engineer: hey I made this cool thing that can help people in public safety roles process information and make decisions more efficiently! It gives false positives but you save more time than it takes less time to weed through them.

Someone nearby: well what if they use it to replace human thinking instead of augment it?

Engineer: well they would be ridiculous. Nobody would ever think that’s a good idea.

Marketing Team: it seems like this lands best when positioning it as a decision-making tool. Let’s get some metrics on how much faster it is at making decisions than people are.

Sales Rep: ok, Captain, let’s dive into our flagship product, DecisionMaker Pro, the totally automated security monitoring agent…

::6 months later—some kid is being held at gunpoint over snacks.::

Re: Armed police swarm student after AI mistakes bag of Doritos for a weapon

#229

Post by avidiax »

omnipresent12 wrote: post #101
neilv wrote: post #91

> Omnilert later admitted the incident was a “false positive” but claimed the system “functioned as intended,” saying its purpose is to “prioritize safety and awareness through rapid human verification.” It just created a situation in which a bunch of people with guns were told that some teen had a gun. That's a very unsafe situation that the system created, out of nothing. And some teen may be traumatized. Again, un…

https://www2.ljworld.com/news/schools/2025/aug/07/lawrence-s... Another false positive by one of these leading content filters schools use - the kid said something stupid in a group chat and an AI reported it to the school, and the school contacted the police. The kid was arrested, stripped searched, and held for 24 hours without access to their parents or counsel. They ultimately had to spend time in probation, a fu…

https://archive.is/DYPBL

> Gaggle’s CEO, Jeff Patterson, said in an interview that the school system did not use Gaggle the way it is intended. The purpose is to find early warning signs and intervene before problems escalate to law enforcement, he said. “I wish that was treated as a teachable moment, not a law enforcement moment,” said Patterson.

It's entirely predictable that schools will call law enforcement for many of these detections. You can't sell to schools that have "zero tolerance" policies and pretend that your product won't trigger those policies.

Re: Armed police swarm student after AI mistakes bag of Doritos for a weapon

#230

Post by chillingeffect »

b00ty4breakfast wrote: post #221
neilv wrote: post #91

> Omnilert later admitted the incident was a “false positive” but claimed the system “functioned as intended,” saying its purpose is to “prioritize safety and awareness through rapid human verification.” It just created a situation in which a bunch of people with guns were told that some teen had a gun. That's a very unsafe situation that the system created, out of nothing. And some teen may be traumatized. Again, un…

>...its purpose is to “prioritize safety and awareness through rapid human verification. Oh look, a corporation refusing to take responsibility for literally anything. How passe.

The invention of the corporation is virtually to eliminate responsibility/culpability from any individual.

Human car crash? Human punishment. Corporate-owned car crash? A fine which reduces salaries some negligible percent.

Post reply on HN