Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Armed police swarm student after AI mistakes bag of Doritos for a weapon

dexerto.com

Post reply on HN
Print view View original

301–310 of 468 posts

Re: Armed police swarm student after AI mistakes bag of Doritos for a weapon

#301

Post by casey2 »

DrewADesign wrote: post #228
neilv wrote: post #91

> Omnilert later admitted the incident was a “false positive” but claimed the system “functioned as intended,” saying its purpose is to “prioritize safety and awareness through rapid human verification.” It just created a situation in which a bunch of people with guns were told that some teen had a gun. That's a very unsafe situation that the system created, out of nothing. And some teen may be traumatized. Again, un…

Engineer : hey I made this cool thing that can help people in public safety roles process information and make decisions more efficiently! It gives false positives but you save more time than it takes less time to weed through them. Someone nearby : well what if they use it to replace human thinking instead of augment it? Engineer : well they would be ridiculous. Nobody would ever think that’s a good idea. Marketing…

Nice fantasy, but the reality is that the "people in public safety roles" love using flimsy pretenses to harass and abuse vulnerable populations. I wish it was just overeager sales and marketing, but you're view of humanity is way too naive especially as masked thugs are disappearing people in the street as we type.

Re: Armed police swarm student after AI mistakes bag of Doritos for a weapon

#302

Post by grafmax »

zugi wrote: post #295
avidiax wrote: post #229

Earlier quoted context omitted.

https://archive.is/DYPBL > Gaggle’s CEO, Jeff Patterson, said in an interview that the school system did not use Gaggle the way it is intended. The purpose is to find early warning signs and intervene before problems escalate to law enforcement, he said. “I wish that was treated as a teachable moment, not a law enforcement moment,” said Patterson. It's entirely predictable that schools will call law enforcement for m…

Exactly. In a saner world, we could use fallible AI to call attention to possible concerns that a human could then analyze and make an appropriate judgment call on. But alas, we don't live in that world. We live in a world where there will be firings, civil, and even criminal liability for those who make wrong judgments. If the AI says "possible gun", the human running things who alerts a SWAT team faces all upside a…

At least the current moment, the increasing turn to using autonomous weaponry against one’s citizens - I don’t think it says so much about humanity so much as the US. I think US foreign policy is a disaster but turning the AI-powered military against the citizenry does look like it’s going to be quite successful, presumably because the US leadership is fighting an enemy incapable of defending itself. I think it’s unsustainable though economically speaking. AI won’t actually create value once it’s a commodity itself (since a true commodity has its value baked into its price). Rates of profit will continue to fall. The ruling class will become increasingly desperate in its search for growth. Eventually an economy that resorts to techno-fascism implodes. (Not before things turning quite ugly of course.)

Re: Armed police swarm student after AI mistakes bag of Doritos for a weapon

#303

Post by A4ET8a8uTh0_v2 »

tacticus wrote: post #261
janalsncm wrote: post #148

Earlier quoted context omitted.

In any system, there are false positives and false negatives. In some situations (like a high recall disease detection) false negatives are much worse than false positives, because the cost of a false positive is a more rigorous screening. But in this case both are bad. If it was a false negative students might need therapy for a more tragic reason. Aside from improving the quality of the detection model, we should t…

> But in this case both are bad. If it was a false negative students might need therapy for a more tragic reason. Given the probability of police officers in the USA taking any action as hostile and then ending up shooting him a false positive here is the same as swatting someone. The system here sent the police off to kill someone.

Yep. Think of it as the new exciting version of swatting. Naturally, one will still need to figure out common ways to force a specific misattribution, but, sadly, I think there will be people working on it ( if there aren't already ).

Re: Armed police swarm student after AI mistakes bag of Doritos for a weapon

#304

Post by CGamesPlay »

nomel wrote: post #225
xbar wrote: post #208

Earlier quoted context omitted.

Charge the superintendent with swatting. Decision-maker accountability is the only thing that halts bad decision-making.

> Charge the superintendent with swatting. This assumes no human verification of the flagged video. Maybe the bag DID look like a gun. We'll never know, because modern journalism has no interest in such things. They obtained the required emotional quotes and moved on.

Human verified the video -> human was the decision-maker. No human verified the video -> Human who gave a blank check to the AI system was the decision-maker. It's not really about the quality of journalism, here.

Re: Armed police swarm student after AI mistakes bag of Doritos for a weapon

#305

Post by ncr100 »

Sincere, and snarky summary:

"Omnilert" .. "You Have 10 Seconds To Comply"

-now targeting Black children!

Q: What was the name of the Google AI Ethicist who was fired by Google for raising the concern that AI overwhelmingly negatively framed non-white humans as threats .. Timnit Gebru

https://en.wikipedia.org/wiki/Timnit_Gebru#Exit_from_Google

We, as technologists, ARE NOT DOING BETTER. We must do better, and we are not on the "DOING BETTER" trajectory.

We talk about these "incidents" with breathless, "Wwwwellll if we just train our AI better ..." and the tragedies keep rolling.

Q2: Which of you has had a half dozen Squad Cars with Armed Police roll up on you, and treat you like you were a School Shooter? Not me, and I may reasonably assume it's because I am white, however I do eat Doritos.

Re: Armed police swarm student after AI mistakes bag of Doritos for a weapon

#306

Post by nomel »

CGamesPlay wrote: post #304
nomel wrote: post #225

Earlier quoted context omitted.

> Charge the superintendent with swatting. This assumes no human verification of the flagged video. Maybe the bag DID look like a gun. We'll never know, because modern journalism has no interest in such things. They obtained the required emotional quotes and moved on.

Human verified the video -> human was the decision-maker. No human verified the video -> Human who gave a blank check to the AI system was the decision-maker. It's not really about the quality of journalism, here.

Please provide the quote from the story that says which of those is the case.

Re: Armed police swarm student after AI mistakes bag of Doritos for a weapon

#307

Post by FireBeyond »

MisterTea wrote: post #199
adaml_623 wrote: post #195

Earlier quoted context omitted.

This is a really good question. Sadly the answer is that they think it's how the system is meant to work. Well that seems to be the answer that I see coming from police spokespeople

Its likely procedure that they have to follow (see my other post in this thread.) I hate to say this but I get it. Imagine a scenario happens where they decide "sounds phony. stand down." only for it to be real and people are hurt/killed because the "cops ignored our pleas for help and did nothing." which would be a horrible mistake they could be liable for, never mind the media circus and PR damage. So they treat al…

I'm a paramedic, who has personally attended a swatting call where every single detail was so egregiously wrong, but police still went in, no-knock, causing thousands of dollars damage, that, to be clear, they have absolutely zero liability for, but thankfully no injuries.

"I can see them in the upstairs window" - of a single story home.

"The house is red brick" - it was dark grey wood.

"No cars in the driveway" - there was two.

Cops still said "hmm, still could be legit" and battered down the front door, deployed flashbangs.

Re: Armed police swarm student after AI mistakes bag of Doritos for a weapon

#308

Post by dfxm12 »

lelandfe wrote: post #182
janalsncm wrote: post #148

Earlier quoted context omitted.

In any system, there are false positives and false negatives. In some situations (like a high recall disease detection) false negatives are much worse than false positives, because the cost of a false positive is a more rigorous screening. But in this case both are bad. If it was a false negative students might need therapy for a more tragic reason. Aside from improving the quality of the detection model, we should t…

I was swatted once. Girlfriend's house. Someone called 911 and said they'd seen me kill a neighbor, drag their body into the house, and was now holding my gf's family hostage. We answered the screams at the door to guns pointed at our faces, and countless cops. It was explained to us that this was the restrained version. We got a knock. Unfortunately, I understand why these responses can't be neutered too much. You j…

It doesn't make sense. If you were holding people hostage, you'd have demands for their release. Windows could be peeked into. If you dragged a dead body into a house, there'd be evidence of that.

Re: Armed police swarm student after AI mistakes bag of Doritos for a weapon

#309

Post by RajT88 »

omnipresent12 wrote: post #101
neilv wrote: post #91

> Omnilert later admitted the incident was a “false positive” but claimed the system “functioned as intended,” saying its purpose is to “prioritize safety and awareness through rapid human verification.” It just created a situation in which a bunch of people with guns were told that some teen had a gun. That's a very unsafe situation that the system created, out of nothing. And some teen may be traumatized. Again, un…

https://www2.ljworld.com/news/schools/2025/aug/07/lawrence-s... Another false positive by one of these leading content filters schools use - the kid said something stupid in a group chat and an AI reported it to the school, and the school contacted the police. The kid was arrested, stripped searched, and held for 24 hours without access to their parents or counsel. They ultimately had to spend time in probation, a fu…

> This is a paid addon, though

Holy shitballs. In my experience such pain addons have very cheap labor attached to them, certainly not what you would expect based on the sales pitch.

Re: Armed police swarm student after AI mistakes bag of Doritos for a weapon

#310

Post by nomel »

joe_the_user wrote: post #276
nomel wrote: post #225

Earlier quoted context omitted.

> Charge the superintendent with swatting. This assumes no human verification of the flagged video. Maybe the bag DID look like a gun. We'll never know, because modern journalism has no interest in such things. They obtained the required emotional quotes and moved on.

So, are you implying that if humans surveil kids at random and call the SWAT team if a frame in a video seems to imply one kid has a gun, that then it's all OK? Those journalists, just trying to get (unjustified, dude, unjustified!!) emotes from kids being mistakenly held at gun point, boy they are terrible.... They're just covering up how necessary those mistakes are in our pursuit of teh crime...

If security sees someone carrying a gun in surveillance video, on a gun free campus, and policy verify it, then yes, that's justified, by all aspects of the law. There are countless examples of surveillance of illegal activity resulting in police action.

Are you suggesting it's not?

Post reply on HN