Live data from Hacker News

The NSA’s machine learning algorithm may be killing thousands of innocent people

arstechnica.co.uk

21–30 of 221 posts

Re: The NSA’s machine learning algorithm may be killing thousands of innocent people

#21
Reminds me of this dialog from Spectre (http://www.imdb.com/title/tt2379713/):

> Have you ever had to kill a man, Max? Have you? To pull that trigger, you have to be sure. Yes, you investigate, analyze, assess, target. And then you have to look him in the eye. And you make the call. And all the drones, bugs, cameras, transcripts, all the surveillance in the world can't tell you what to do next.

Re: The NSA’s machine learning algorithm may be killing thousands of innocent people

#22
The thing that stood out for me is "Somewhere between 2,500 and 4,000 people have been killed by drone strikes in Pakistan since 2004".

WHAT!? That is so wrong! This stinks and we're making fuss about mathematics. Just read that sentence again.

Re: The NSA’s machine learning algorithm may be killing thousands of innocent people

#23
post #18
post #13

Big data analysis + mass surveillance is a frightening prospect. Of course you can train software to look for 'terrorists', but you could also train it to look for: - whistleblowers - minority groups (e.g. gay people, particular religious beliefs, political affiliation) - political dissidents - journalists whose behaviour changes - personal vulnerabilities (affairs, mental health issues etc) Think what authoritarian…

I would call it "mass discrimination". Because, in principle, that is what this amounts to: you're using a few traits of a person to generalize into a whole group. I find it strange that nobody seems to identify this. It is almost as if, because it is done on a big computer and no human is involved in the processing, suddenly it is not discriminating anymore. But humans programmed these computers in the first place.

This has been identified, posted to HN, and discussed.[1]

[1] https://news.ycombinator.com/item?id=10872852

Re: The NSA’s machine learning algorithm may be killing thousands of innocent people

#24
post #15
post #9

This story is stupid. I'm sorry. No fan of the NSA, but the premise behind it is completely ridiculous. There is zero evidence of the repeatably asserted idea that the list this tool generates is any kind of kill list. It's a tool that generates indicators of people that may be worth looking at when trying to find couriers. That's a very specific subgroup of terrorists, and I find it entirely unsurprising that a jour…

You may of course be right, I would find it surprising in the extreme if people were killed based purely on output of an algorithm. However we have no assurance that this tool was not used to generate kill targets without any human input. The NSA operates without any meaningful oversight and if they decided to use algorithms to generate kill targets, they would go right ahead and do it.

> However we have no assurance that this tool was not used to generate kill targets without any human input.

That's absurd. We also have no assurance that the NSA isn't communicating with aliens to generate kill lists of people who will oppose the alien invasion of 2021, though we do have common sense to filter out such absurd scenarios. "We have no assurance that this isn't so" isn't an argument.

Re: The NSA’s machine learning algorithm may be killing thousands of innocent people

#25

They literally named it Skynet. They have an evil sense of humor. Actually using machine learning to detect terrorists isn't a terrible idea. But you are going to get an error rate, and probably a high one in the noisy real world. Maybe only 50% of the people you detect are actually terrorists. Maybe it's even worse than that. We can't even test it because there is no validation set and unreliable labels. The reasona…

> But you are going to get an error rate, and probably a high one in the noisy real world. ... The reasonable thing to do with that information, would be to surveil them further

It is even worse than that. Due to the base rate fallacy (https://en.wikipedia.org/wiki/Base_rate_fallacy) the 'machine' is going to have a huge false positive rate, so without that additional surveillance, most will not be guilty of what they are accused. But that surveillance takes money and resources, just the thing the 'machine' system is designed to save...

Re: The NSA’s machine learning algorithm may be killing thousands of innocent people

#26
post #9

This story is stupid. I'm sorry. No fan of the NSA, but the premise behind it is completely ridiculous. There is zero evidence of the repeatably asserted idea that the list this tool generates is any kind of kill list. It's a tool that generates indicators of people that may be worth looking at when trying to find couriers. That's a very specific subgroup of terrorists, and I find it entirely unsurprising that a jour…

How can we review these lists/tools and reach our own conclusions? References please.

I'm completely unconvinced that it isn't reasonable for an agency to keep lists and perhaps tools like this secret.

Police don't let criminals know when they are being watched, and that is generally seen as reasonable.

Abuse is a problem. Not sure what the solution is, but I'm not sure making this type of list public is the solution.

Re: The NSA’s machine learning algorithm may be killing thousands of innocent people

#27

They literally named it Skynet. They have an evil sense of humor. Actually using machine learning to detect terrorists isn't a terrible idea. But you are going to get an error rate, and probably a high one in the noisy real world. Maybe only 50% of the people you detect are actually terrorists. Maybe it's even worse than that. We can't even test it because there is no validation set and unreliable labels. The reasona…

If you are going to put surveillance on potential terrorists, you should also be putting surveillance on people not suspected of being terrorists, so you can judge the rate at which your surveillance personnel falsely accuse people being terrorists based on having been selected for surveillance.

Single or double blind that process.

Re: The NSA’s machine learning algorithm may be killing thousands of innocent people

#28
post #10
post #6

Earlier quoted context omitted.

If they did not, what could Pakistan do about it? The U.S would do so anyway covertly, and the Pakistani government would be seen to be weak, since they were powerless to stop it. So the Pakistani government 'allows' it to at least appear in control.

They have nukes. They could easily institute a defense doctrine of nuclear retaliation tomorrow, and the terror would stop. But the reason they don't is that they are bought and paid for, both as a country via foreign aid and as individual government members via bribes (or "speaker fees", as you guys call those now).

SAVAGE

Re: The NSA’s machine learning algorithm may be killing thousands of innocent people

#29
post #13

Big data analysis + mass surveillance is a frightening prospect. Of course you can train software to look for 'terrorists', but you could also train it to look for: - whistleblowers - minority groups (e.g. gay people, particular religious beliefs, political affiliation) - political dissidents - journalists whose behaviour changes - personal vulnerabilities (affairs, mental health issues etc) Think what authoritarian…

To be honest, being someone interested in lots of things, just based on my browser history you could also put me in the terrorist box.

Re: The NSA’s machine learning algorithm may be killing thousands of innocent people

#30

They literally named it Skynet. They have an evil sense of humor. Actually using machine learning to detect terrorists isn't a terrible idea. But you are going to get an error rate, and probably a high one in the noisy real world. Maybe only 50% of the people you detect are actually terrorists. Maybe it's even worse than that. We can't even test it because there is no validation set and unreliable labels. The reasona…

> The reasonable thing to do with that information, would be to surveil them further, search their house, or arrest them. Not assassinate them without a trial.

Which seems to be exactly the case. Ahmad Zaidan, for example, wasn't assassinated. In fact even if the algorithm was 100% correct - it makes no sense to assassinate people solely based on whether they are terrorists or not. A lot more can be gained from surveilling some of them, etc'.

So there is really nothing at all to suggest that those algorithms alone are used to create kill lists.

Post reply on HN