Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Viral Tweet About Apple Card Leads to Goldman Sachs Probe

bloomberg.com

Post reply on HN
Print view View original

41–50 of 180 posts

Re: Viral Tweet About Apple Card Leads to Goldman Sachs Probe

#41

Post by themgt »

jacquesm wrote: post #26
themgt wrote: post #17

I would take DHH's interpretation of this with a large grain of salt. He seems to assume that living in a "community property state" will be taken into account (presumably then considering his wife to have parity with his income/score) by the credit limit algorithm, but we have no way to know that that's the case. He also adds "It gets even worse. Even when she pays off her ridiculously low limit in full, the card wo…

It won't have that line in their code. What it very likely will have is some Bayesian algo or maybe if they're really high tech some ML black box that will have gender as one of its inputs. And it probably shouldn't have that input. The whole idea behind decisions like these is that they have to be explainable and ML especially does not lend itself well for that.

Here's a thread on Apple support[1], with many reporting similar experience and an anecdote going the opposite way:

I’ve been having the same issue since Monday 10/14. I paid my balance in full, money was taken from my Chase account right away and cleared next day, but my Apple Card available balance hasn’t updated. I’ve also gotten the same response from support. It could take several days. Funny thing I paid my wife’s card off a few weeks ago and it updated within seconds, same bank account and everything......

[1] https://discussions.apple.com/thread/250676909?page=2

Re: Viral Tweet About Apple Card Leads to Goldman Sachs Probe

#42

Post by zrail »

themgt wrote: post #17

I would take DHH's interpretation of this with a large grain of salt. He seems to assume that living in a "community property state" will be taken into account (presumably then considering his wife to have parity with his income/score) by the credit limit algorithm, but we have no way to know that that's the case. He also adds "It gets even worse. Even when she pays off her ridiculously low limit in full, the card wo…

We don’t know, and that’s the problem. Machine learning models are entirely dependent on their training data. If the data is biased the model will be too.

Re: Viral Tweet About Apple Card Leads to Goldman Sachs Probe

#43

Post by toomuchtodo »

xenihn wrote: post #39
tootie wrote: post #25

One data point really isn't enough to draw a conclusion and I actually doubt it's true. GS is extremely scrupulous when it comes to avoiding liability and they probably put a ton of diligence into their risk model. It's possible there's some emergent gender bias but we'd need a better show of proof.

What we really need is the ability to force organizations to expose their models.

To regulators, sure. Entirely reasonable. To your average Joe? No. This is already done to prevent redlining when originating mortgages.

Disclaimer: Work in financial services in risk management, interface with regulators. Opinions are my own.

Re: Viral Tweet About Apple Card Leads to Goldman Sachs Probe

#44

Post by resonantjacket5 »

xenihn wrote: post #39
tootie wrote: post #25

One data point really isn't enough to draw a conclusion and I actually doubt it's true. GS is extremely scrupulous when it comes to avoiding liability and they probably put a ton of diligence into their risk model. It's possible there's some emergent gender bias but we'd need a better show of proof.

What we really need is the ability to force organizations to expose their models.

Problem is once a model is exposed it can be easily gamed

Re: Viral Tweet About Apple Card Leads to Goldman Sachs Probe

#45

Post by setpatchaddress »

function_seven wrote: post #19
rndgermandude wrote: post #13

Other than him being very outraged and claiming gender-based discrimination again and again, is there actually anything to suggest the algorithm is biased on the gender axis, as opposed to any other axis? Just to be clear: I am not dismissing the possibility that the algorithm (meaning the training data, really) is gender biased, it just isn't clear from what I have seen in the tweet storm that this is necessarily th…

> is there actually anything to suggest the algorithm is biased on the gender axis, as opposed to any other axis? None of us are allowed to know that. That's the big problem here. They delegate the decision to a black box that cannot be questioned. When the process is set up like that, I think it's fair to assume the worst case scenarios and put the burden on the company to prove otherwise. Adverse inference is a sen…

it’s notable that a lot of mid-20th-century fear surrounding the increased use of computers was centered around this exact scenario: a black box, the judgement of which cannot be questioned, deciding your fate, with no recourse.

We all scoffed at this for a long time. ML makes it real, apparently.

Re: Viral Tweet About Apple Card Leads to Goldman Sachs Probe

#50

Post by jacquesm »

matthewmacleod wrote: post #36
jacquesm wrote: post #26

Earlier quoted context omitted.

It won't have that line in their code. What it very likely will have is some Bayesian algo or maybe if they're really high tech some ML black box that will have gender as one of its inputs. And it probably shouldn't have that input. The whole idea behind decisions like these is that they have to be explainable and ML especially does not lend itself well for that.

Note that an ML box doesn’t even need gender as an input. Name alone probably gets you 90% of the way there.

True, you could have an internal table that maps name to gender with a very high probability of being right.
Post reply on HN