Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Notes on AI Bias

ben-evans.com

Post reply on HN
Print view View original

11–20 of 126 posts

Re: Notes on AI Bias

#11

Post by TheCoelacanth »

FakeComments wrote: post #6
fvdessen wrote: post #2

> Since Amazon’s current employee base skews male, the examples of ‘successful hires’ also, mechanistically, skewed male and so, therefore, did this system’s selection of resumés. Amazon spotted this and the system was never put into production. Couldn't they have retrained the system with a 50/50 mix of males / females resumes ? Or restrict the use of the algorithm to sort male resumes ? Or maybe resumes don't actua…

If the only problem is that it mixes up male and female resumes, why not have it sort each into two piles and have a human recruiter perform the merge by picking the better applicant from the pair on top, repeatedly? I have the suspicion that Amazons AI was factually correct, but politically wrong.

I highly doubt that the preference for male candidates was the only problem with the AI. The preference for candidates with ice hockey on their resume almost certainly also would have resulted in a preference for white candidates.

Re: Notes on AI Bias

#12

Post by munchbunny »

fvdessen wrote: post #2

> Since Amazon’s current employee base skews male, the examples of ‘successful hires’ also, mechanistically, skewed male and so, therefore, did this system’s selection of resumés. Amazon spotted this and the system was never put into production. Couldn't they have retrained the system with a 50/50 mix of males / females resumes ? Or restrict the use of the algorithm to sort male resumes ? Or maybe resumes don't actua…

I think this retelling of the story is over-simplified. It's a compelling story, but I don't know any competent engineers who give up on a whole project because of one setback. If this system never saw production use, it was because it's still not ready, or there were many other issues that aren't mentioned that led the team to give up, or because political winds shifted. Amazon is famous for killing projects quickly.

Re: Notes on AI Bias

#13

Post by TheCoelacanth »

eanzenberg wrote: post #9

>>Now, suppose that 75% of the bad turbines use a Siemens sensor and only 12% of the good turbines use one (and suppose this has no connection to the failure). The system will build a model to spot turbines with Siemens sensors. Oops. Given a statistically large enough sample, 2 outcomes: 1) The Siemens sensor actually is at fault. 2) The Siemens sensor is a part of a larger system, which is different in non-Siemens…

Only if your test data is free of sample bias.

Given how incredibly hard it is to avoid sample bias, you can't take it for granted that your training data doesn't have any sample bias.

Re: Notes on AI Bias

#14

Post by twa927 »

> Until about 2013, If you wanted to make a software system that could, say, recognise a cat in a photo, you would write logical steps. You’d make something that looked for edges in an image, and an eye detector, and a texture analyser for fur, and try to count legs, and so on, and you’d bolt them all together...

I'm doing a lot of such algorithms (well, not for images). Does someone know if such algorithms have a name? I'm calling it "heuristics" and I think it falls under "AI".

Re: Notes on AI Bias

#15

Post by DuskStar »

TheCoelacanth wrote: post #13
eanzenberg wrote: post #9

>>Now, suppose that 75% of the bad turbines use a Siemens sensor and only 12% of the good turbines use one (and suppose this has no connection to the failure). The system will build a model to spot turbines with Siemens sensors. Oops. Given a statistically large enough sample, 2 outcomes: 1) The Siemens sensor actually is at fault. 2) The Siemens sensor is a part of a larger system, which is different in non-Siemens…

Only if your test data is free of sample bias. Given how incredibly hard it is to avoid sample bias, you can't take it for granted that your training data doesn't have any sample bias.

If the sample is "all the gas turbines I own", I don't particularly CARE about the bias...

Re: Notes on AI Bias

#16

Post by FakeComments »

TheCoelacanth wrote: post #11
FakeComments wrote: post #6

Earlier quoted context omitted.

If the only problem is that it mixes up male and female resumes, why not have it sort each into two piles and have a human recruiter perform the merge by picking the better applicant from the pair on top, repeatedly? I have the suspicion that Amazons AI was factually correct, but politically wrong.

I highly doubt that the preference for male candidates was the only problem with the AI. The preference for candidates with ice hockey on their resume almost certainly also would have resulted in a preference for white candidates.

Sure, my point is why does that matter?

The correlation with ice hockey could be a career relevant detail because of correlation with discipline and pain tolerance. If it tracks all such signals, I don’t see the problem.

That it also has a circumstantial correlation with races isn’t inherently problematic — it could be that different groups of people are differently qualified.

There seems to be a position that culture is uncorrelated to job success, and that’s just nonsense. When people can show me these biases are correlated to race or sex once controlling for culture, then we have a problem.

Re: Notes on AI Bias

#17

Post by chobeat »

twa927 wrote: post #14

> Until about 2013, If you wanted to make a software system that could, say, recognise a cat in a photo, you would write logical steps. You’d make something that looked for edges in an image, and an eye detector, and a texture analyser for fur, and try to count legs, and so on, and you’d bolt them all together... I'm doing a lot of such algorithms (well, not for images). Does someone know if such algorithms have a na…

I call these approaches: "there must be OpenCV in there somewhere"

Re: Notes on AI Bias

#18

Post by manfredo »

fvdessen wrote: post #2

> Since Amazon’s current employee base skews male, the examples of ‘successful hires’ also, mechanistically, skewed male and so, therefore, did this system’s selection of resumés. Amazon spotted this and the system was never put into production. Couldn't they have retrained the system with a 50/50 mix of males / females resumes ? Or restrict the use of the algorithm to sort male resumes ? Or maybe resumes don't actua…

The whole problem with allegations of AI bias is that people often point to disparities of outcome as proof of bias. The reality is that there are plenty of disparities on outcome regardless of bias, and the allegations of bias and attempt to rectify the alleged bias is another vector for the introduction of bias.

Re: Notes on AI Bias

#19

Post by throwawaymath »

eanzenberg wrote: post #9

>>Now, suppose that 75% of the bad turbines use a Siemens sensor and only 12% of the good turbines use one (and suppose this has no connection to the failure). The system will build a model to spot turbines with Siemens sensors. Oops. Given a statistically large enough sample, 2 outcomes: 1) The Siemens sensor actually is at fault. 2) The Siemens sensor is a part of a larger system, which is different in non-Siemens…

No, that's incorrect. Note the part of your quote which says, "and suppose this has no connection to the failure."

The point is the Siemens sensor is a superfluous correlation with turbine failure, because the underlying dataset is biased towards Siemens sensors. The scenario suggested by the author is one in which your turbine failure dataset does not match reality.

No amount of sample enlargement will correct sample bias. You have a variable which is disproportionately represented in your underlying dataset despite being independent from a collection of variables correlated to failure, and the algorithm is learning that one instead.

Real world ways this is plausible and cannot be corrected by increased sampling:

1. Your telemetry data is accurate, but your logging service providing that data is faulty and only consumes data from a subset of meaningful publishers.

2. Whoever provided this dataset fat fingered a SQL query which joined too few tables including the sensor vendors, but correctly returned only the failing turbines.

3. Your data has (unnormalized) duplicates, because more than one system is providing telemetry data for Siemens sensors without the older systems being retired.

4. You use mostly Siemens sensors, and simply didn't correct for this in your sample.

Re: Notes on AI Bias

#20

Post by chobeat »

DuskStar wrote: post #15
TheCoelacanth wrote: post #13

Earlier quoted context omitted.

Only if your test data is free of sample bias. Given how incredibly hard it is to avoid sample bias, you can't take it for granted that your training data doesn't have any sample bias.

If the sample is "all the gas turbines I own", I don't particularly CARE about the bias...

you should, because you might make worse decisions for the business, for the system or for the people that are impacted by the system. If you don't have the right data to decide, don't decide using the data.
Post reply on HN