Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How to recognize AI snake oil [pdf]

cs.princeton.edu

Post reply on HN
Print view View original

311–320 of 364 posts

Re: How to recognize AI snake oil [pdf]

#311

Post by SuoDuanDao »

xmprt wrote: post #267
gpm wrote: post #213

Earlier quoted context omitted.

Snaring customers too. I swear, people are obsessed with "machine learning" even when the domain really isn't suited for it.

I wonder if people are becoming more savvy and anti-ML after all the issues people have with Facebook and Google collecting a lot more data than people are comfortable with.

I suspect what people dislike is big data. Anecdotally, I'd love to hear about a success where some lone genius codes a better tool using publicly available datasets. I don't like the idea of BigCorp using their larger servers to dominate the space.

Re: How to recognize AI snake oil [pdf]

#312

Post by iamsb »

dcminter wrote: post #290
iamsb wrote: post #240

Earlier quoted context omitted.

Speaking of jobs and interviews. I am yet to find a job board which does not show JavaScript jobs when searching for Java jobs. Some of them claim to use AI. :)

In fairness I have this issue with a number of human intelligence based (allegedly) recruiters.

Of course, that's where machines gets it from as well. Your training set is usually which candidates were shortlisted by recruiters. At least that is what I have seen people do who are built one of these AI based recruitment systems. When I asked them are they not worried about such bad quality data, their answer is - humans know the best. Ah well, then why are you building this again?

Re: How to recognize AI snake oil [pdf]

#313

Post by mikorym »

reggieband wrote: post #58

I don't have time to read the entire paper but I would like to share an anecdote. I worked at a company with a well staffed/funded machine learning team. They were in charge of recommendation systems - think along the lines of youtube up next videos. My team wanted better recommendations (really, less editorial intensive) so the ML team spent weeks crafting 12 or more variants of their recommendation system for our c…

IMH (and biased) O, a lot of great coders are implimentors, or let's say applied computers scientists.

That is great for building incredible open source software and a lot of other things that I would not be able to do given a 1000 years. However (again IMHBO) a specific ML and any other specific application of stastistics or mathematics becomes really tricky once your use case is explicitly defined.

You then need intimate and deep knowledge of the tools that you are using (e.g.: Should I even use NN? Should I even use genetic algorithms? Should I even use x?) but ML for most people is shorthand for NN and its variants or maybe shorthand for something else specific rather than in principle.

A well aimed shot at PCA [1] can often solve your problem. Or at least, tell you what the problem looks like. This is just an example, but IMHBO people waste their time learning ML and not learning mathematics and statistics.

IMHBO I still think that self-driving cars can be solved by defining a list of 1000 or so rules, by hand, by humans, and by consensus. The computer vision part is the ML part.

[1] https://en.wikipedia.org/wiki/Principal_component_analysis

Re: How to recognize AI snake oil [pdf]

#314

Post by sfifs »

Jestar342 wrote: post #296
andybak wrote: post #202

Earlier quoted context omitted.

I just spent 15m on Amazon trying to prod the recommendation algorithm into finding something I actually wanted to buy so I could get above the "free delivery threshold". Think about that. I wanted to spend money. I wasn't too fussy what it was. Amazon has a decade of my purchasing and browsing history. And they still failed.

Amazon infuriates me. I regularly buy gadget-y bits like electronic components and peripherals. Probably at least once a month. Never see adverts for similar. That ONE TIME I buy a unicorn dress for my 2 year old daughter? That's a lifetime of unicorn related merchandise adverts and recommendations for you!

This is basically what comes from a mindset increasingly common among ML practitioners of abdicating thinking and assuming "the machine will find what features are important". They throw a junkyard full of features at the algorithm (or even worse automated feature generation). These days at least once a fortnight I get the opportunity to show folks how if only they thought for 10-15 mins or simply charted their data in a few different cuts before modeling, how much better they could have done :-)

Re: How to recognize AI snake oil [pdf]

#315

Post by xorfish »

visarga wrote: post #281
chimi wrote: post #176

Earlier quoted context omitted.

I agree and rather than post a sibling response, I'll add that I think it's necessary today , simply because we don't have AGI, yet . And also point out that we are talking about determining if AI is snake oil or not. There may be some scenarios where we can teach a computer to do something we can't teach a human to do, I can't think of any off the top of my head, but if we can't, then I'm going to be super doubtful…

Same for AlphaGo, but it only works when you have access to cheap simulation (breakout being a game easy to run, Go being just a board). It doesn't work as well in situations where you don't see the full state of the system, or where there is randomness.

AlphaGo did pretty well in Starcraft 2. Even though it is still pretty far from the best players in the world.

Re: How to recognize AI snake oil [pdf]

#316

Post by mark-r »

chimi wrote: post #165
mark-r wrote: post #147

Earlier quoted context omitted.

> You can't teach a human how to do something when the methodology to do that is the student trying something and the teacher saying "Yes" or "No". Strangely, I recently had to complete a cognitive test that was essentially that process. I was given a series of pages, each of which had a number of shapes and a multiple choice answer. I was told whether I chose the correct answer, then the page was flipped to the next…

You're touching on the "difficulty" in verbalizing it. I see what you mean, because you did learn that the heuristic was changing with just a yes or no. I said you can't teach that way, but you clearly learned that way, so I wasn't exactly correct, but I'm not practically wrong either still I don't think. I wonder, how would an AI perform on the same test. What is the mathematical minimum number of questions on such…

I'm sure the test was a standard with a name, but I was never told. It was a small part of a 3 hour ordeal, evaluating my healing progress since suffering a brain injury in March.

I would agree that it's a very inefficient way of teaching something. It gave me an unexpected insight into machine learning though.

I'm sure the test was designed so that picking the same answer each time or picking one at random would result in a fail.

Re: How to recognize AI snake oil [pdf]

#317

Post by mark-r »

gbear605 wrote: post #200
chimi wrote: post #165

Earlier quoted context omitted.

You're touching on the "difficulty" in verbalizing it. I see what you mean, because you did learn that the heuristic was changing with just a yes or no. I said you can't teach that way, but you clearly learned that way, so I wasn't exactly correct, but I'm not practically wrong either still I don't think. I wonder, how would an AI perform on the same test. What is the mathematical minimum number of questions on such…

Doesn’t it depend on what you mean by guarantee? The test can’t get 100% certainty, since theoretically you could be flipping a coin each time and miraculously getting it right, for 1000 times in a row. The chance of that is minuscule (1/2^1000), but it’s nonzero. So we’d have to define a cutoff point for guaranteed. The one used generally in many sciences is 1/20 chance (p = 0.05), so that seems like a plausible one…

There were more than two possible answers to choose from on each page, so the odds of being right were considerably lower.

Re: How to recognize AI snake oil [pdf]

#318

Post by SAI_Peregrinus »

maxlamb wrote: post #44
buboard wrote: post #14

Earlier quoted context omitted.

You should emphasize that this is Organic AI. It's low carbon and overall greener.

I think simply OI (Organic Intelligence) would the more appropriate term since it's no longer artificial ;)

Eh, every existing instance was created by some people working together.

Re: How to recognize AI snake oil [pdf]

#319

Post by dgellow »

madrox wrote: post #140
reggieband wrote: post #58

I don't have time to read the entire paper but I would like to share an anecdote. I worked at a company with a well staffed/funded machine learning team. They were in charge of recommendation systems - think along the lines of youtube up next videos. My team wanted better recommendations (really, less editorial intensive) so the ML team spent weeks crafting 12 or more variants of their recommendation system for our c…

I had a similar experience at one point. A team put energy into building a recommendation system, and were able to demonstrate that the "Recommended for you" content performed better than all other content editorial. After getting challenged a bit, though, turns anything performs better when put under the header "Recommended for you."

Well, that’s a good lesson to learn, just put everything under “recommended for you” header! :)

Re: How to recognize AI snake oil [pdf]

#320

Post by JoeAltmaier »

nicholas73 wrote: post #112
JoeAltmaier wrote: post #105

Earlier quoted context omitted.

The San Jose Mercury News had a weather-forecasting contest. It was won one year by a 12-year-old, who's algorithm was "The weather tomorrow will be the same as the weather today". A kind of AI I guess.

To be fair San Jose weather is heavily biased towards sameness.

Yes, but why did all the 'smart' people using models and training networks get a worse result? A condemnation of modelling, to be sure.
Post reply on HN