Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Armed police swarm student after AI mistakes bag of Doritos for a weapon

dexerto.com

Post reply on HN
Print view View original

261–270 of 468 posts

Re: Armed police swarm student after AI mistakes bag of Doritos for a weapon

#261

Post by tacticus »

janalsncm wrote: post #148
neilv wrote: post #91

> Omnilert later admitted the incident was a “false positive” but claimed the system “functioned as intended,” saying its purpose is to “prioritize safety and awareness through rapid human verification.” It just created a situation in which a bunch of people with guns were told that some teen had a gun. That's a very unsafe situation that the system created, out of nothing. And some teen may be traumatized. Again, un…

In any system, there are false positives and false negatives. In some situations (like a high recall disease detection) false negatives are much worse than false positives, because the cost of a false positive is a more rigorous screening. But in this case both are bad. If it was a false negative students might need therapy for a more tragic reason. Aside from improving the quality of the detection model, we should t…

> But in this case both are bad. If it was a false negative students might need therapy for a more tragic reason.

Given the probability of police officers in the USA taking any action as hostile and then ending up shooting him a false positive here is the same as swatting someone.

The system here sent the police off to kill someone.

Re: Armed police swarm student after AI mistakes bag of Doritos for a weapon

#262

Post by kergonath »

underlipton wrote: post #223
MisterTea wrote: post #199

Earlier quoted context omitted.

Its likely procedure that they have to follow (see my other post in this thread.) I hate to say this but I get it. Imagine a scenario happens where they decide "sounds phony. stand down." only for it to be real and people are hurt/killed because the "cops ignored our pleas for help and did nothing." which would be a horrible mistake they could be liable for, never mind the media circus and PR damage. So they treat al…

Cops don't have a duty to protect people, so "cops ignored our pleas for help and did nothing" is a-ok, no liability (thank you, qualified immunity). They very much do not treat all scenarios as real; they go gung-ho when they want to and hang back for a few hours "assessing the situation" when they don't.

> they go gung-ho when they want to and hang back for a few hours "assessing the situation" when they don't.

Yeah. They were happy to take their sweet time assessing everything safely outside the buildings at Uvalde.

Re: Armed police swarm student after AI mistakes bag of Doritos for a weapon

#263

Post by akoboldfrying »

Zigurd wrote: post #152
akoboldfrying wrote: post #125

Earlier quoted context omitted.

> Make them pay money for false positives instead of direct support and counselling. Agreed. > This technology is not ready for production No one wants stuff like this to happen, but nearly all technologies have risks. I don't consider that a single false positive outweighs all of its benefits; it would depend on the rates of false and true positives, and the (subjective) value of each (both high in this case, though…

In the US cops kill more people than terrorists. As long as you quantifying values take that into account.

I get that people are uncomfortable with explicit quantification of stuff like this, but removing the explicitness doesn't remove the quantification, it just makes it implicit. If, say, we allow people to drive cars even though car accidents kill n people each year, then we are implicitly quantifying that the value of the extra productivity society gets by being able to get places quickly in a car is worth the deaths of those people.

In your example, if terrorists were the only people killing people in the US, and police (a) were the only means of stopping them, and (b) did not benefit society in any other way, the equation would be simple: get rid of the police. There wouldn't need to be any value judgements, because everything cancels out. But in practice it's not that easy, since the vast majority of killings in the US occur at the hands of people who are neither police nor terrorists, and police play a role in reducing those killings too.

Re: Armed police swarm student after AI mistakes bag of Doritos for a weapon

#264

Post by GuinansEyebrows »

akoboldfrying wrote: post #248
GuinansEyebrows wrote: post #151

Earlier quoted context omitted.

> This argument can be made about almost every technology, including fire, electricity, and even the base "technology" of just letting a person watch a camera feed. huh, i can think of at least one recent example of a popular figure using this kind of argument to extreme self-detriment.

Is HN really this ready to dive into obvious logical fallacies? My original comment, currently sitting at -4, has so far attracted one guilt-by-association plus implied-threat combo, and no other replies. To remind readers: My horrifying proposal was to measure both the risks and the benefits of things . If anyone genuinely thinks measuring the risks and benefits of things is a bad idea, or that it is in general a go…

sorry for being glib; it was low hanging fruit. my actual point should have been more clearly stated: measuring risk/benefit is really complicated because there's almost never a direct comparison to be made when balancing profit, operational excellence and safety.

Re: Armed police swarm student after AI mistakes bag of Doritos for a weapon

#265

Post by Aeolun »

nkrisc wrote: post #181
janalsncm wrote: post #148

Earlier quoted context omitted.

In any system, there are false positives and false negatives. In some situations (like a high recall disease detection) false negatives are much worse than false positives, because the cost of a false positive is a more rigorous screening. But in this case both are bad. If it was a false negative students might need therapy for a more tragic reason. Aside from improving the quality of the detection model, we should t…

In this case false positives are far, far worse than false negatives. A false negative in this system does not mean a tragedy will occur, because there are many other preventative measures in place. And never mind the fact that this country refuses to even address the primary cause of gun violence in the first place: the ubiquity of guns in our society. So systems like this is what we end up with when we ignore to ad…

I tend to categorize these under a dutch idiom which I can’t describe, but which is abundantly clear in pictorial form:

https://klimapedia.nl/wp-content/uploads/2020/01/Dweilen_met...

Re: Armed police swarm student after AI mistakes bag of Doritos for a weapon

#266

Post by anigbrowl »

xp84 wrote: post #234
beloch wrote: post #231

Systems like this need to report confidence in their assertions. e.g. Not "this student has a gun" but "this model says the student has a gun with a probability of 60%". If an AI can't quantify it's degree of confidence, it shouldn't be used for this sort of thing.

Even better, share the frame(s) that the guess was drawn from with a human for verification before triggering ANYTHING. How much trouble could that possibly be? How many "guns" is this thing detecting in a day across all sites? I doubt more than a couple or we'd have heard about tons of incidents, false positives or not. I wanna see the frames too.

I don't find that especially good as a sole remedy, because lots of people are stupid. If they see a green outline box overlaid on a video image with the label 'gun', many many people will just respond to the label instead of looking at the underlying image and trying to make a decision. Probability and validation history need to be built into the product so that there are audit logs that can be pored over and challenged. Bad human decision-making, which is rampant, is always smoothed over with justifications like 'I was concerned for everyone's safety', and usually treated in isolation rather than assessed longitudinally.

Re: Armed police swarm student after AI mistakes bag of Doritos for a weapon

#267

Post by rglover »

This may be mean, but we should really be careful about just handing AI over to technically illiterate people. They're far more likely to blind trust the LLM/AI output than someone who may be more experienced and take a beat. AI in an agentic-state society (what we have in America at least) is an absolute ticking time bomb. Honestly, this is what AI safety teams should be concentrated on: making sure people who think the computer is infallible understand that, no, it isn't, and you shouldn't just assume what it tells you is correct.

Re: Armed police swarm student after AI mistakes bag of Doritos for a weapon

#268

Post by Aeolun »

pixelready wrote: post #258
Lammy wrote: post #226

Earlier quoted context omitted.

> the primary cause of gun violence in the first place: the ubiquity of guns in our society I would have gone with “a normalized sense of hopelessness and indignity which causes people to feel like violence is the only way they can have any agency in life” considering “gun” is the adjective and “violence” is the actual thing you're talking about.

Both are true. The underlying oppressive, lonely, pro-bullying culture creates the tension. The proliferation of high lethality weapons makes it more likely that tension will eventually release in the form of a mass tragedy. Improvement in either area would be a net positive for society. Improvement in both areas is ideal but solving proliferation seems a lot more straightforward than fixing the generally miserable s…

I think there’s probably some correlation between ‘generally miserable society’ and ‘we think it’s ok to have children surveiled by AI’

Re: Armed police swarm student after AI mistakes bag of Doritos for a weapon

#269

Post by joe_the_user »

tartoran wrote: post #88

"“We understand how upsetting this was for the individual that was searched as well as the other students who witnessed the incident,” the principal wrote. “Our counselors will provide direct support to the students who were involved.”" Make them pay money for false positives instead of direct support and counselling. This technology is not ready for production, it should be in a lab not in public buildings such as s…

I assume they were provide gift cards good for psychotherapy sessions.
Post reply on HN