81–90 of 91 posts
How does "simple AI" compare with an old fashioned small table lookup and a few "if" statements?
If I run N different machine learning models over the same data, and each has some random error in fitting the objective function, then I pick the one which matches the validation data best, isn't there a danger of picking the one which was "luckiest" with the random errors? Presumably for large N that's a real problem? How do people account for that?
For large N sizes the models will tend to converge on the same logic. It’s actually much more of a problem with small samples. Simpler models like linear regressions and decision trees could even be deterministic.
Earlier quoted context omitted.
I am unclear as to whether what you are saying is that they need data science to justify their projects because when they say that an AI told them that they need $X extra for the project it carries more weight than when they say they did the analysis by hand or if the data science actually provided value in this case.
No. They need AI in their projects to receive management buy in (and funding) because management swallowed the hook and now wants everything AI & data driven (even if it makes no sense) to be able to brag to other managers from the competition and other industries that they now use AI and are a data driven company.
This headline and article are horrible and misrepresent the problem and the outcome. The paper is about the manual process of re-assigning a credit score on a scale of 1 to 15 based on other customer criteria. Really the fact that this process exists at all shows that their initial credit scoring approach is flawed or too simplistic. The argument of "just replace it with an if statement" does not hold up in this scen…
Plus, systemic risk of a repeatable exploitation is more likely without humans in the loop. Making a bad loan for $1M is bad, but if “attackers” can repeatedly prove until they get a bad risk $1B loan, it becomes business shattering.
No bank in any credible jurisdiction will have an automated system approving 1bn USD equivalent loans any time soon. A typical system would be loans up to a certain amount can get approved automatically, up to $xM by a credit officer, up to $yM by a senior credit officer, anything over that by the credit committee. Regulators push back very hard on automated decisionmaking for large loans particularly because of "default correllation skew"[1] problems as were revealed in the 2008 crisis. Relatively few bad decisions on big loans can push a bank into difficulties if they are not well-capitalized. This is particularly a problem for automated decision-making because as loans get larger they also get more idiosyncratic and therefore it's much harder to fit a model with confidence because there simply aren't enough data points.[2]
[1] Often credit quality for a group of loans rises because of idiosyncratic factors but deteriorates together, so as loans become more risky in an adverse economic environment the correllation of the default probability between the loans goes up. An intuitive way of thinking about this is in housing loans. If 3 or 4 of your neighbours default on their loan, property prices on your street will go down (because the banks will be trying to sell all those houses at once), making it much more likely you and the rest of the residents will default too.
[2] Say I'm trying to approve a 200k loan to expand a pizza restaurant. If I'm in a big bank I have hundreds of similar loans to use as data points for pricing and risk. If I'm trying to approve a 200M loan to build a luxury hotel complex that includes 5 restaurants, accommodation, retail etc it is completely a one-off. Even if I'm the largest commercial lender this loan will be unique in my portfolio. I will have many other large loans but there will be lots of idiosyncratic factors that make them different.
Earlier quoted context omitted.
Plus, systemic risk of a repeatable exploitation is more likely without humans in the loop. Making a bad loan for $1M is bad, but if “attackers” can repeatedly prove until they get a bad risk $1B loan, it becomes business shattering.
Do they remove the human in the loop? That doesn't seem like a smart idea. A model would be good just for suggestions. Humans are both biased and with high variance (not to mention corruptible), but the algorithm can benefit from much better scrutiny and ensure uniform application of the criteria. If a human overrides, then they got to have a good reason.
[1] https://ec.europa.eu/info/law/law-topic/data-protection/refo...
If you want to impress me, tell me also the results of these: - Performance against a fair dice - Performance against a group of humans trained to predict bank managers' loan decisions
minimum_credit_score = Loan.joins(:borrower).where("loans.default = true").average("borrowers.credit_score") if loan_applicant.credit_score > minimum_credit_score decision = "approve" else decision = "reject" end
How does "simple AI" compare with an old fashioned small table lookup and a few "if" statements?
Toward Data Science has a neat example [1], scroll to the end to see the sample "scorecard" which can be implemented as lookup tables and ifs. Example shows how the customer gets certain points depending on the age group, home ownership and income group. Sum up these points and you get the total score, which then translates to probability of getting your money back from that customer.
[1] https://towardsdatascience.com/intro-to-credit-scorecard-9af...
Earlier quoted context omitted.
Since you have industry experience how would you have banks rectify this perceived injustice? I can’t conceive of an underwriting process worthy of the name that isn’t going to consider those borrowers with collateral more creditworthy than those without.
For one, overdraft fees should only be charged once per calendar month, not for every overage. Also, the burden on the bank for an overdraft is near zero, so why is it $35 for the customer? Perhaps a fee of only $5 would be less regressive?
The author looks at empirical evidence (experiments) and shows that you can take experts' predictions, use a super-dumb statistical model of what they do, and then outperform the experts' judgments with a simple linear model!