Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The NSA’s machine learning algorithm may be killing thousands of innocent people

arstechnica.co.uk

Post reply on HN
Print view View original

91–100 of 221 posts

Re: The NSA’s machine learning algorithm may be killing thousands of innocent people

#91

Post by tgr35a »

nl wrote: post #9

This story is stupid. I'm sorry. No fan of the NSA, but the premise behind it is completely ridiculous. There is zero evidence of the repeatably asserted idea that the list this tool generates is any kind of kill list. It's a tool that generates indicators of people that may be worth looking at when trying to find couriers. That's a very specific subgroup of terrorists, and I find it entirely unsurprising that a jour…

https://www.youtube.com/watch?v=UdQiz0Vavmc "We kill people based on metadata" - Michael Hayden, ex NSA-Chief.

Re: The NSA’s machine learning algorithm may be killing thousands of innocent people

#92

Post by mike_hearn »

pavelrub wrote: post #30
Houshalter wrote: post #7

They literally named it Skynet. They have an evil sense of humor. Actually using machine learning to detect terrorists isn't a terrible idea. But you are going to get an error rate, and probably a high one in the noisy real world. Maybe only 50% of the people you detect are actually terrorists. Maybe it's even worse than that. We can't even test it because there is no validation set and unreliable labels. The reasona…

> The reasonable thing to do with that information, would be to surveil them further, search their house, or arrest them. Not assassinate them without a trial. Which seems to be exactly the case. Ahmad Zaidan, for example, wasn't assassinated. In fact even if the algorithm was 100% correct - it makes no sense to assassinate people solely based on whether they are terrorists or not. A lot more can be gained from surve…

Yeah but nothing going on in the national security state makes much sense. The CIA drone program needs targets. It devours them. When they ran out of real intelligence they started killing people they didn't even know the name of because they "looked like terrorists", check out signature strikes.

Those drones are going to be tasked no matter what. If the quality of the input intelligence is crap, well, good thing it's all top secret right!

Re: The NSA’s machine learning algorithm may be killing thousands of innocent people

#93

Post by mike_hearn »

Swizec wrote: post #17
agd wrote: post #13

Big data analysis + mass surveillance is a frightening prospect. Of course you can train software to look for 'terrorists', but you could also train it to look for: - whistleblowers - minority groups (e.g. gay people, particular religious beliefs, political affiliation) - political dissidents - journalists whose behaviour changes - personal vulnerabilities (affairs, mental health issues etc) Think what authoritarian…

>- minority groups (e.g. gay people So there was this infamous incident of a Netflix machine learning competition for movie recommendations ... Guess what somebody was able to do.

I can guess what you must mean but I can't find any references about it. Got a link?

Re: The NSA’s machine learning algorithm may be killing thousands of innocent people

#94

Post by pps43 »

qeabc wrote: post #62
Houshalter wrote: post #7

They literally named it Skynet. They have an evil sense of humor. Actually using machine learning to detect terrorists isn't a terrible idea. But you are going to get an error rate, and probably a high one in the noisy real world. Maybe only 50% of the people you detect are actually terrorists. Maybe it's even worse than that. We can't even test it because there is no validation set and unreliable labels. The reasona…

Perhaps someone who knows the topic can say whether this mathematical phenomena of "adversarial examples" for neural networks can be translated to random forests: http://www.kdnuggets.com/2015/07/deep-learning-adversarial-e... Barely perceptible changes to an image cause it to be misclassified by neural networks which never saw the image before (this is important, because it rules out simple overfitting). As a non-ex…

It would be difficult to create such persuasive adversarial examples in this case. Humans can recognize images well, so it's obvious for us that those adversarial examples are nowhere near what neural net thinks they are. It would not be as obvious with a list of phone calls.

Re: The NSA’s machine learning algorithm may be killing thousands of innocent people

#95

Post by DKnol »

botterworkshop wrote: post #79

It's ignorant to assume any targets revealed by this program wouldn't be given a human intelligence analyst to verify accuracy before risking millions of dollars on a predator strike and the potential risk it was bad information. Enough said.

https://theintercept.com/2014/02/10/the-nsas-secret-role/

Re: The NSA’s machine learning algorithm may be killing thousands of innocent people

#97

Post by mike_hearn »

nl wrote: post #48
titzer wrote: post #33

Earlier quoted context omitted.

I'll bite. The difference is that communicating with aliens requires positing the existence of aliens, whereas simply generating a kill list is already something we know they do, though we don't know how it works. It isn't "common sense" to filter out entirely obvious actions the NSA would take, and given the scale of disclosures from the Snowden files you ought to have already expanded your imagination away from "th…

The fact that 2,500 to 4,000 people have been killed in Pakistan according to a completely opaque process for classifying targets and blasting them from the sky--a process that operates entirely outside any law--ought to be considered absurd, shocking, frightening, and soul-crushingly inhuman. But it's not. We're arguing about with each other about bullshit. I'm incredibly appalled by that. That's why stories like th…

* who will then produce evidence to show that this particular program does exactly what it tries to to*

No they won't, because they've been challenged to do such things repeatedly in the past and always failed.

These people live in a foreign country and it's not like the US Govt dispatches a bunch of detectives and lawyers based on the results of this ML model. Get real. The intelligence is handed off to the CIA without revealing how it's generated, the CIA then says "we got a list of terrorist couriers from the NSA, let's go get em" and boom, off it goes.

This is all incredibly well documented.

There is simply no mathematical way the program described in the article can be accurate, that's what the entire article is about. So I don't see why you have such profound faith in them. It's quite clear they're a bunch of maths geeks who have a single hammer and will use it to hammer any US foreign policy problem regardless of how much it resembles a nail or not.

Re: The NSA’s machine learning algorithm may be killing thousands of innocent people

#98

Post by moyix »

pwg wrote: post #25
Houshalter wrote: post #7

They literally named it Skynet. They have an evil sense of humor. Actually using machine learning to detect terrorists isn't a terrible idea. But you are going to get an error rate, and probably a high one in the noisy real world. Maybe only 50% of the people you detect are actually terrorists. Maybe it's even worse than that. We can't even test it because there is no validation set and unreliable labels. The reasona…

> But you are going to get an error rate, and probably a high one in the noisy real world. ... The reasonable thing to do with that information, would be to surveil them further It is even worse than that. Due to the base rate fallacy ( https://en.wikipedia.org/wiki/Base_rate_fallacy ) the 'machine' is going to have a huge false positive rate, so without that additional surveillance, most will not be guilty of what t…

Yes. I feel like anyone considering such a system should be forced to read "The Base-Rate Fallacy and its Implications for the Difficulty of Intrusion Detection":

http://www.raid-symposium.org/raid99/PAPERS/Axelsson.pdf

Re: The NSA’s machine learning algorithm may be killing thousands of innocent people

#99

Post by mike_hearn »

avar wrote: post #39
nl wrote: post #9

This story is stupid. I'm sorry. No fan of the NSA, but the premise behind it is completely ridiculous. There is zero evidence of the repeatably asserted idea that the list this tool generates is any kind of kill list. It's a tool that generates indicators of people that may be worth looking at when trying to find couriers. That's a very specific subgroup of terrorists, and I find it entirely unsurprising that a jour…

Furthermore, even if the algorithm was generating kill lists directly there seems to be the implicit assumption that that would be worse than having some human "yes-men" who are directly beholden to orders of their immediate superiors and surely care more about their career advancement than bombing random Pakistanis in charge of the process. It's the same fallacy as being overly paranoid about the safety of self-driv…

Informers. Of which there are a limited number ... and of course they are themselves not entirely reliable which is why virtually all developed western countries have scrapped the death penalty.

Re: The NSA’s machine learning algorithm may be killing thousands of innocent people

#100

Post by eli_gottlieb »

Houshalter wrote: post #7

They literally named it Skynet. They have an evil sense of humor. Actually using machine learning to detect terrorists isn't a terrible idea. But you are going to get an error rate, and probably a high one in the noisy real world. Maybe only 50% of the people you detect are actually terrorists. Maybe it's even worse than that. We can't even test it because there is no validation set and unreliable labels. The reasona…

Wait wait wait. So they're killing people based on a model that fails Statistics 101!? No accounting for base rates? No cross-validation!?

God fucking damnit, why the fuck does our government think killing should be cheaper and easier than healing!? We put new medical treatments through decades of difficult scientific tests, but have none at all for strictly invalid ways to kill people?

Post reply on HN