Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How to recognize AI snake oil [pdf]

cs.princeton.edu

Post reply on HN
Print view View original

51–60 of 364 posts

Re: How to recognize AI snake oil [pdf]

#51

Post by drongoking »

formalsystem wrote: post #15

Over the years my heuristic has turned into: "Did the team formulate their problem as a supervised learning problem?" - If not it's probably BS. In longform if anyone is interested https://medium.com/@marksaroufim/can-deep-learning-solve-my-... EDIT: I would consider autoencoders, word2vec, Reinforcement Learning examples of turning a different problem into a supervised learning problem EDIT 2: Social functions like…

You miss the point of the slides. His point isn't about supervised vs unsupervised, it's about the general areas where AI seems to excel and fail. It's being used to predict social outcomes where it does very poorly, may be inscrutable, and is unaccountable to the public.

Your examples (deep learning applied to perception) are what he argues AI is generally good for.

Re: How to recognize AI snake oil [pdf]

#52

Post by gok »

formalsystem wrote: post #15

Over the years my heuristic has turned into: "Did the team formulate their problem as a supervised learning problem?" - If not it's probably BS. In longform if anyone is interested https://medium.com/@marksaroufim/can-deep-learning-solve-my-... EDIT: I would consider autoencoders, word2vec, Reinforcement Learning examples of turning a different problem into a supervised learning problem EDIT 2: Social functions like…

I don't follow that. The recidivism predictor was supervised. Conversely, AlphaZero is unsupervised and certainly not BS.

Re: How to recognize AI snake oil [pdf]

#53

Post by rasmi »

account73466 wrote: post #22
rasmi wrote: post #8

Earlier quoted context omitted.

Not to indulge the troll, but Arvind Narayanan is an (associate) professor of CS at Princeton and is one of the foremost researchers in the field on topics of ML/data privacy and ethics [0]. His papers/talks/tweets regularly attract attention on HN [1]. That you're judging the talk based on which conferences the author hasn't published in says more about your ignorance of the STS field than it does about the author's…

self-censored, fanboys don't buy it

Congratulations on your success! I'm actually familiar with the author's past talks and research, and am not just assuming he's competent because he lists his affiliation with Princeton.

I encourage you to familiarize yourself with the field of socio-technical systems. It is related to (but not the same as) "ML/DL", and it is important to know about if you are doing research in CS. A good place to start is the FAT* conference [0] (which was previously a workshop at NeurIPS).

Regarding manual scoring: The author cites this study [1] and specifically says: "This is a falsifiable claim. Of course, I’m willing to change my mind or add appropriate caveats to the claim if contrary evidence comes to light. But given the evidence so far, this seems the most prudent view." so by all means, do reach out to him with better evidence.

[0]: https://fatconference.org/2019/program.html

[1]: https://arxiv.org/abs/1702.04690

Re: How to recognize AI snake oil [pdf]

#54

Post by eloff »

buboard wrote: post #14
avip wrote: post #10

My company is sourcing AI from MTurk. It's actually cheaper than running fat GPU model training instances. The network learns fast and adapts well to changes in inputs. I envision the sticker "human inside" strapped on our algorithms.

You should emphasize that this is Organic AI. It's low carbon and overall greener.

You mean high carbon, low silicon? Because humans usually have a higher carbon footprint than computers, it takes a lot of computers to match one human. Plus we're made of carbon.

Re: How to recognize AI snake oil [pdf]

#55

Post by fyp »

formalsystem wrote: post #15

Over the years my heuristic has turned into: "Did the team formulate their problem as a supervised learning problem?" - If not it's probably BS. In longform if anyone is interested https://medium.com/@marksaroufim/can-deep-learning-solve-my-... EDIT: I would consider autoencoders, word2vec, Reinforcement Learning examples of turning a different problem into a supervised learning problem EDIT 2: Social functions like…

There's a lot of gray area between unsupervised and supervised learning. For example self-supervised learning: https://www.facebook.com/722677142/posts/10155934004262143/

Re: How to recognize AI snake oil [pdf]

#56

Post by gok »

I really wish we could stop using AI or ML for things in the "predicting social outcomes" category. Naming them more like "computational astrology" or "machine alchemy" would be a better fit.

Re: How to recognize AI snake oil [pdf]

#57

Post by confrnz »

formalsystem wrote: post #15

Over the years my heuristic has turned into: "Did the team formulate their problem as a supervised learning problem?" - If not it's probably BS. In longform if anyone is interested https://medium.com/@marksaroufim/can-deep-learning-solve-my-... EDIT: I would consider autoencoders, word2vec, Reinforcement Learning examples of turning a different problem into a supervised learning problem EDIT 2: Social functions like…

Ironically, the algorithm you pose in that comment itself, is a BS algorithm in it of itself.

"Formulate the problem as X" - what is your input for how a problem is formulated? That you personally like how it was formulated?

"Probably," - OK, so you assign probability scores? Or do you mean, "likelihood based upon my guess?"

Finally, how do you measure performance? Your own assessment of how good you were at it?

Re: How to recognize AI snake oil [pdf]

#58

Post by reggieband »

I don't have time to read the entire paper but I would like to share an anecdote. I worked at a company with a well staffed/funded machine learning team. They were in charge of recommendation systems - think along the lines of youtube up next videos. My team wanted better recommendations (really, less editorial intensive) so the ML team spent weeks crafting 12 or more variants of their recommendation system for our content. We then ran that system for months in an A/B testing system that judged user behaviour by several KPI. The result was all variants performed equally well within statistically insignificant bounds. The best performing variant happened to be random.

Talking to other groups that had gone through the exact same process our results were pretty typical. These guys were all very intelligent and the code and systems they had implemented were pretty impressive. I'm guessing the system they built would have cost a few million dollars if built from scratch. We did use this "AI/ML" in our marketing so maybe it was payed for by increased sales through use of buzz words. But my experience was that in most limited use cases the technology was ineffective.

Re: How to recognize AI snake oil [pdf]

#59

Post by xeRTRex »

formalsystem wrote: post #15

Over the years my heuristic has turned into: "Did the team formulate their problem as a supervised learning problem?" - If not it's probably BS. In longform if anyone is interested https://medium.com/@marksaroufim/can-deep-learning-solve-my-... EDIT: I would consider autoencoders, word2vec, Reinforcement Learning examples of turning a different problem into a supervised learning problem EDIT 2: Social functions like…

Auto-encoders have been more successful in fraud and anomaly detection then supervised methods. For the uninitiated: the basic concept is to reduce the feature space (i.e. the things you know) to a lower dimensional space, then decode back into the original space. When enough differences arise between the original and reconstructed variables, the event may be flagged for a human to review (or some triage process).

Re: How to recognize AI snake oil [pdf]

#60

Post by hcarvalhoalves »

account73466 wrote: post #22
rasmi wrote: post #8

Earlier quoted context omitted.

Not to indulge the troll, but Arvind Narayanan is an (associate) professor of CS at Princeton and is one of the foremost researchers in the field on topics of ML/data privacy and ethics [0]. His papers/talks/tweets regularly attract attention on HN [1]. That you're judging the talk based on which conferences the author hasn't published in says more about your ignorance of the STS field than it does about the author's…

self-censored, fanboys don't buy it

You seem to be knowledgeable on the matter then. Why hide behind a throw-away account and the ad hominem against the author though? You could articulate better your perspective, we would appreciate. (I hope you're doing okay)
Post reply on HN