Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Machine learning’s crumbling foundations

pluralistic.net

Post reply on HN
Print view View original

51–60 of 106 posts

Re: Machine learning’s crumbling foundations

#51

Post by AlbertCory »

iamstupidsimple wrote: post #45
dmix wrote: post #34

You could use this article's underlying thesis to explain why a lot of tech companies fail as well. Google Health is a good example of failing to appreciate specialization and domain-expertise. Trying to draw value from broad generic data collection when IRL it requires vertical-focused domain-oriented collection and analysis to really draw value. Funnelling everything into a giant pool of data only had so much value…

> Google Health is a good example of failing to appreciate specialization and domain-expertise. Trying to draw value from broad generic data collection when IRL it requires vertical-focused domain-oriented collection and analysis to really draw value. I'm not sure I agree with this statement. From what I've heard, Google Health employed a huge team of doctors and they were included through the entire feature developm…

Algorithm:

1) Decide to take over Domain X.

2) Hire a bunch of people from Domain X. Don't hire anyone who doesn't agree that you can take over X.

3) Make them report to the people whose idea it was in the first place. If they say "Hey, maybe this wasn't such a great idea" then push them out, as an example to the rest.

4) FAIL.

Note that #1 is the key. The decision to do it precedes the hiring.

Re: Machine learning’s crumbling foundations

#52

Post by amelius »

simonw wrote: post #11

My favourite example of bad data in for machine learning is the tragic tale of Scots Wikipedia: https://www.theguardian.com/uk-news/2020/aug/26/shock-an-aw-... It turned out an enthusiastic but misguided US teenager who didn't actually know the Scots language was responsible for most of the entries on it... and a bunch of natural language machine learning models had already been trained on it.

Reminds me of: https://en.wikipedia.org/wiki/English_as_She_Is_Spoke

Re: Machine learning’s crumbling foundations

#53

Post by maxerickson »

xyzzy21 wrote: post #50
axelroze wrote: post #10

It's a structural issue caused by the way wealth creation works for majority of people in tech. Job hopping, trendy frameworks in CV, "high-impact" projects done ASAP, etc. No one wants to do boring, slow pace work with lots of planning, reflection and introspection. And why would they do it? These kind of jobs are usually worst paid. We, the practitioners, have every economic incentive to go the other route. The pro…

Sadly PCR tests for COVID also test positive for flu and half a dozen other causes. That's why CDC/FDA are seeking proposals for a new test that actually works! https://www.cdc.gov/csels/dls/locs/2021/07-21-2021-lab-alert...

You've fallen for the internet. Please restart and try again.

https://www.reuters.com/article/factcheck-covid19-pcr-test-i...

Re: Machine learning’s crumbling foundations

#54

Post by robbedpeter »

qayxc wrote: post #30
notanzaiiswear wrote: post #13

It sounds like cherry picking bad examples to me. Likewise you could say "programming's foundations are crumbling" by citing all sorts of programming projects that use bad or faulty code. Meanwhile, speech recognition seems to work extremely well by now (I am a little bit older, so I remember when it didn't work so well). I am also not aware of any real world cases of AI being used to detect Corona, so that seems to…

> Meanwhile, speech recognition seems to work extremely well by now (I am a little bit older, so I remember when it didn't work so well). *provided you speak English or Mandarin, the former preferably of a continental US variety It's astonishing how bad things get again once you mix in an accent, local dialect (e.g. Swiss German) or a less frequently spoken language (like Croatian).

Why would you expect dialects with vastly fewer training examples to be on par with the most widely spoken languages? It's a simple matter of available data, and the state of the art architectures operate on a paradigm that scales quality of the model to quantity of training data.

If you want better speech recognition for Swiss-German, then record and transcribe hundreds of thousands of hours or whatever level of parity you want to achieve with recognition.

It's not "astonishing" at all. Models won't generalize well unless they have sufficient data, so to achieve multi accent functionality, we need lots more high quality data. Or we need better architectures, so identifying where models fail and engineering a better architecture could be a breakthrough. The shortcomings are not surprising or mysterious at all, it's simply a function of the nature of these algorithms.

Re: Machine learning’s crumbling foundations

#55

Post by killjoywashere »

I ask the following as someone who builds and tests models and also annotates data as a domain expert. Is labeling really undervalued by society? Or just by VCs?

I mean, if society depends more on the labeler (e.g. radiologists) why should society reward people for trying to replace the radiologists, regardless of the data quality?

From a societal perspective where human factors scientists tell us that we need people to actually be employed to achieve a sense of self-worth and happiness, shouldn’t we punish labelers who might otherwise only enrich the capitalists and undermine the health of the nation’s workforces, and thus the wellbeing of the nation as a whole? Did we learn nothing from the underemployed, disaffected, demoralized, suicidally depressed Trump electorate?

The Trump presidency may be a hot mess from which the country may never recover, but are these not the lessons that we ostensibly learned, that were the topic of millions of gallons of ink between 2016 and 2018?

Re: Machine learning’s crumbling foundations

#56

Post by zmmmmm »

6gvONxR4sf7o wrote: post #27

> One common failure mode? Treating data that was known to be of poor quality as if it was reliable because good data was not available… they also use the poor quality data to assess the resulting models. This drives me nuts. Spend $10k getting high quality data and throw a simple model at it? Nah, let’s spend a month of time from someone making $400k/yr for less trustworthy results. And on the blogosphere it’s even…

I feel like its even worse than just resorting to bad data when good data isn't available: the field of deep learning has cultivated the perception that it's robust to bad data as one of its hallmarks.

That is, you can pump relatively raw data into it and it will self-select features and then self-regulate their use so therefore most of the initial steps of data cleaning, feature selection etc are not necessary, or require less expertise. This is now spilling over into general ML so that when quacks assert that their model just magically overcomes these things and people actually believe it.

Re: Machine learning’s crumbling foundations

#57

Post by notanzaiiswear »

alpaca128 wrote: post #43
notanzaiiswear wrote: post #32

Earlier quoted context omitted.

Nevertheless, the huge jump is from "does not work at all" to "it works". It seems likely that the technology that worked for English will also work for many other languages. As for Chinese, it is also pretty amazing that you can visit a Chinese website, click "translate" in your browser's menu bar, and get a reasonably readable translated version. I wonder if people just take too many things for granted. Or internet…

> It seems likely that the technology that worked for English will also work for many other languages. It won't for the foreseeable future. Not for technical reasons; it's just that other languages are usually not handled correctly because most companies think they can just use the exact same approach as in English and they're done. Until they realise that non-English native speakers also use English words and abbrev…

Maybe people using Google should start to apply some common sense and not believe everything at face value. Nevertheless, the examples you cite are extremes that affect only few people. So you would rather have no internet search engines at all, so that those problems could be avoided?

Isn't that a bit like saying cars are crap because people die in accidents? Maybe there are just upsides and downsides to most new technologies, and if the upsides outweigh the downsides by far, people will go for it?

As for human intelligence, I am not convinced humans would necessarily fare better at such tasks. I mean they fall for the "same name, same person" fallacy.

Re: Machine learning’s crumbling foundations

#58

Post by notanzaiiswear »

AlbertCory wrote: post #49
notanzaiiswear wrote: post #31

Earlier quoted context omitted.

What do you mean? At least from the point of view of the end user, apps seem to become better over time.

I was tempted to just downvote this, but I thought I'd reply instead: No, they do not. An existing version of an app may get better over time, but unfortunately it then gets replaced with a different version, which starts from the position of extreme bugginess. In the case of Microsoft Office apps, for instance, one could easily argue that they are steadily getting worse as more and more features are added. Google Ch…

So why not go back to some old version of it? I don't think "memory consumption" is necessarily a good indicator, because sometimes using more memory is a sign of good optimization.

Also how is the memory consumption if you turn off all modern features?

Re: Machine learning’s crumbling foundations

#59

Post by bsanr2 »

axelroze wrote: post #10

It's a structural issue caused by the way wealth creation works for majority of people in tech. Job hopping, trendy frameworks in CV, "high-impact" projects done ASAP, etc. No one wants to do boring, slow pace work with lots of planning, reflection and introspection. And why would they do it? These kind of jobs are usually worst paid. We, the practitioners, have every economic incentive to go the other route. The pro…

Why would anyone care to fix things? The way they are are perfectly amenable to the blame- and conclusion-laundering many ML clients seek.
Post reply on HN