Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The NSA’s machine learning algorithm may be killing thousands of innocent people

arstechnica.co.uk

Post reply on HN
Print view View original

11–20 of 221 posts

Re: The NSA’s machine learning algorithm may be killing thousands of innocent people

#11

Post by karmacondon »

I was wondering exactly how the NSA trained models to detect terrorists, and was surprised by the level of detail in this article. So the NSA performs classification using a Random Forest and about 80 input features? Huh. That actually sounds a little too similar to a Kaggle contest for my liking, but I wasn't expecting that much quasi-technical information anyway.

It seems a little silly to write this whole article based on a few powerpoint slides from years ago. Even as an amateur practitioner, I can see several obvious things that could be changed about the presented methods. I'm sure what the NSA is doing now in this regard is much further along than what was portrayed in those documents. Whether or not they should be doing it is a discussion for others to have. But it is interesting to learn a little more about the details of truly high stakes machine learning.

Re: The NSA’s machine learning algorithm may be killing thousands of innocent people

#13

Post by agd »

Big data analysis + mass surveillance is a frightening prospect. Of course you can train software to look for 'terrorists', but you could also train it to look for:

- whistleblowers

- minority groups (e.g. gay people, particular religious beliefs, political affiliation)

- political dissidents

- journalists whose behaviour changes

- personal vulnerabilities (affairs, mental health issues etc)

Think what authoritarian governments could (and are) doing with this capability.

Re: The NSA’s machine learning algorithm may be killing thousands of innocent people

#14

Post by fsloth »

Aeolos wrote: post #2

"Obliterated a wedding because people where celebrating by shooting in the air." This is the definition of evil. How and why is the Pakistani government allowing this program to operate within their territories?

My guess would be because Pakistan has had good relations with US since it's independence.

"The United States today provides extensive economic, scientific, and military assistance to Pakistan... and is Pakistan's most generous donor of foreign aid"[1]

[1] https://en.wikipedia.org/wiki/Pakistan%E2%80%93United_States...

Re: The NSA’s machine learning algorithm may be killing thousands of innocent people

#15

Post by SeanDav »

nl wrote: post #9

This story is stupid. I'm sorry. No fan of the NSA, but the premise behind it is completely ridiculous. There is zero evidence of the repeatably asserted idea that the list this tool generates is any kind of kill list. It's a tool that generates indicators of people that may be worth looking at when trying to find couriers. That's a very specific subgroup of terrorists, and I find it entirely unsurprising that a jour…

You may of course be right, I would find it surprising in the extreme if people were killed based purely on output of an algorithm. However we have no assurance that this tool was not used to generate kill targets without any human input. The NSA operates without any meaningful oversight and if they decided to use algorithms to generate kill targets, they would go right ahead and do it.

Re: The NSA’s machine learning algorithm may be killing thousands of innocent people

#17

Post by Swizec »

agd wrote: post #13

Big data analysis + mass surveillance is a frightening prospect. Of course you can train software to look for 'terrorists', but you could also train it to look for: - whistleblowers - minority groups (e.g. gay people, particular religious beliefs, political affiliation) - political dissidents - journalists whose behaviour changes - personal vulnerabilities (affairs, mental health issues etc) Think what authoritarian…

>- minority groups (e.g. gay people

So there was this infamous incident of a Netflix machine learning competition for movie recommendations ...

Guess what somebody was able to do.

Re: The NSA’s machine learning algorithm may be killing thousands of innocent people

#18

Post by amelius »

agd wrote: post #13

Big data analysis + mass surveillance is a frightening prospect. Of course you can train software to look for 'terrorists', but you could also train it to look for: - whistleblowers - minority groups (e.g. gay people, particular religious beliefs, political affiliation) - political dissidents - journalists whose behaviour changes - personal vulnerabilities (affairs, mental health issues etc) Think what authoritarian…

I would call it "mass discrimination". Because, in principle, that is what this amounts to: you're using a few traits of a person to generalize into a whole group.

I find it strange that nobody seems to identify this. It is almost as if, because it is done on a big computer and no human is involved in the processing, suddenly it is not discriminating anymore.

But humans programmed these computers in the first place.

Re: The NSA’s machine learning algorithm may be killing thousands of innocent people

#19

Post by citizensixteen »

nl wrote: post #9

This story is stupid. I'm sorry. No fan of the NSA, but the premise behind it is completely ridiculous. There is zero evidence of the repeatably asserted idea that the list this tool generates is any kind of kill list. It's a tool that generates indicators of people that may be worth looking at when trying to find couriers. That's a very specific subgroup of terrorists, and I find it entirely unsurprising that a jour…

How can we review these lists/tools and reach our own conclusions?

References please.

Re: The NSA’s machine learning algorithm may be killing thousands of innocent people

#20

Post by teekert »

Houshalter wrote: post #7

They literally named it Skynet. They have an evil sense of humor. Actually using machine learning to detect terrorists isn't a terrible idea. But you are going to get an error rate, and probably a high one in the noisy real world. Maybe only 50% of the people you detect are actually terrorists. Maybe it's even worse than that. We can't even test it because there is no validation set and unreliable labels. The reasona…

Given that there are very little real terrorists in the data, any sensitive detection mechanism will need insane specificity or it will give you an enormous amount of false positives.
Post reply on HN