Amazon’s Face Recognition Falsely Matched 28 Members of Congress with Mugshots • Hacker News Board

Re: Amazon’s Face Recognition Falsely Matched 28 Members of Congress with Mugshots

#111

Post by Kalium » Fri, Jul 27, 2018, 3:37 AM UTC

StanislavPetrov wrote: post #110

Kalium wrote: post #87

Earlier quoted context omitted.

They fed it a large number of mugshots. They didn't ask "is this person a criminal". They asked it "does this face match one of the faces you've seen before". You're right that systems have previously learned to infer black -> criminal. It's absolutley possible that could have happened here! Given that not every one flagged is a person of color, there's clearly more at work. Systems like Rekognition let a user specif…

They didn't tell it to use 80%, 80% is the default setting. Anyone who thinks that many law enforcement agencies aren't going to use the default setting ought to go look at their routers and other hardware and see how they are configured.

I understand where you're coming from. They didn't choose that number! It was chosen for them by Amazon! It's on Amazon for not actively forcing users to consider what level of confidence they want to use. Amazon can, should, and I think is perhaps even morally obligated to make customers actively choose a confidence level.

Yet, might there not be other factors to consider? Bear in mind that this isn't pushbutton easy off-the-shelf. Some actual work is required here. There's probably a batch of Python scripts sitting around that did the work. The most charitable assumption is that whoever did this at the ACLU didn't look at how the tool under examination should be used at all and missed everything about confidence levels in whatever few docs they did read. Under the ACM code of ethics, I am comfortable calling that an unethical degree of incompetence and carelessness.

A less charitable interpretation is that they knew and didn't care because it produced the result they wanted. This is similarly unethical, in my mind on a level with the sort of p-hacking prevalent in social sciences.

You're completely right! They just used the default. It's just perhaps possible that there might be other factors worth considering.

Re: Amazon’s Face Recognition Falsely Matched 28 Members of Congress with Mugshots

#112

Post by Kalium » Fri, Jul 27, 2018, 3:43 AM UTC

lawlessone wrote: post #21

cthalupa wrote: post #15

https://www.theverge.com/2018/7/26/17615634/amazon-rekogniti... >Reached by The Verge, an Amazon spokesperson attributed the results to poor calibration. The ACLU’s tests were performed using Rekognition’s default confidence threshold of 80 percent — but Amazon says it recommends at least a 95 percent threshold for law enforcement applications where a false ID might have more significant consequences. Presented witho…

Would that mean 5% could be false positives? That would be a lot.

From the AWS docs:

> A confidence score is a number between 0 and 100 that indicates the probability that a given prediction is correct. In the tropical beach example, if the object and scene detection process returns a confidence score of 99 for the label ‘Water’ and 35 for the label ‘Palm Tree’, then it is more likely that the image contains water but not a palm tree.

> Applications that are very sensitive to detection errors (false positives) should discard results associated with confidence scores below a certain threshold. The optimum threshold depends on the application. In many cases, you will get the best user experience by setting minimum confidence values higher than the default value.

Which I am interpreting as similar to your question, but perhaps should not be understood as a statistical probability.

Re: Amazon’s Face Recognition Falsely Matched 28 Members of Congress with Mugshots

#113

Post by Kalium » Fri, Jul 27, 2018, 3:47 AM UTC

rrego wrote: post #80

andrewguenther wrote: post #76

> People of color were disproportionately falsely matched in our test. They are trying to make this racially charged without giving enough information to verify their claims. If you use a dataset of mugshots, that's statistically going to have more data on people of color. If you have more data on people of color, it is more likely to match people of color. Claiming the algorithm is racist because your data is racist…

Pretty lousy argument. Firstly, the majority of inmates in the US are white (58%). I'd assume mug shot stats are similar, or at the very least not mostly black people. Moreover, if a algorithm enforces bias to the detriment of inoccents, it's a bad algorithm

> Pretty lousy argument. Firstly, the majority of inmates in the US are white (58%). I'd assume mug shot stats are similar, or at the very least not mostly black people.

Personally, I would assume the mugshots are a random sample of a public corpus, and thus likely 35-40% people of color. Which is definitely skewed from the background population. I'm assuming the similarity of 40% is a coincidence.

> Moreover, if a algorithm enforces bias to the detriment of inoccents, it's a bad algorithm

Is it? It sounds like it might be working exactly as it was set up and specified to work. I would call that the fault of the designers and specifiers, rather than the algorithm. It's a good algorithm. It fits its purpose well. Its purposes just happen to be completely evil.

Re: Amazon’s Face Recognition Falsely Matched 28 Members of Congress with Mugshots

#114

Post by otakucode » Sun, Jul 29, 2018, 11:00 AM UTC

zo1 wrote: post #75

otakucode wrote: post #32

Earlier quoted context omitted.

When facial recognition fails, it usually looks pretty weird. Like it will identify faces in the knots and grain patterns of a wooden wall behind somebody. It's really a very ineffective technology and it's disturbing that it gets sold as something you could use to justify locking someone in a cage over. Machines with pareidolia are not good tools.

When people speak of "facial recognition", they speak of faces being compared to other faces. What you're mentioning is face-detection. Are you honestly saying that police departments will have a "criminal database" filled with wooden walls and furniture?

I'm saying that police departments DO have databases filled with images that have wooden walls and such. Not furniture usually, but photos are often taken in sub-optimal circumstances even when the police or others are trained to take the photos properly. I know this is the case because I've worked on biometric systems for the past 18 years, and I have personally worked on a system which was identifying almost every face to a mugshot taken against a wall with wooden paneling and the matching algorithm was keying in on it as a face rather than the human. I'm not speaking theoretically, this is how these systems fail.