Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

You probably don't need AI/ML. You can make do with well written SQL scripts

threadreaderapp.com

Post reply on HN
Print view View original

31–40 of 331 posts

Re: You probably don't need AI/ML. You can make do with well written SQL scripts

#31

Post by threeseed »

rokhayakebe wrote: post #7
kthejoker2 wrote: post #2

As someone who sells both of these services, I can only add that it depends, and if you have a good dataset, it's trivial to write either one. But once you start having to account for noise or seasonality or autoregression or dynamic weights or non linear kernel spaces, pure SQL really starts to fall down on the job.

Curious. OP gave a few examples for Ecommerce where SQL will do fine. Can you give a few where ML will do something otherwise impossible or harder with SQL?

Feature extraction is the most obvious.

I give you a problem e.g. "tell me what factors are influencing a customer's likelihood to leave a bad review". And with ML you can actually produce a list of sorted factors with weighted percentages. You simply can't do that with SQL.

Also remember that SQL is for databases so you often can't do anything algorithmic e.g. K-means clustering, linear regression, random forest, decision trees etc. Where as ML encompasses non-databases as well e.g. from a Kafka stream of numbers predict the next number.

Re: You probably don't need AI/ML. You can make do with well written SQL scripts

#32

Post by dizzystar »

The best is when you ask someone why they want an AI/ML masterwork, they just say it's the future and we don't want to be left behind.

It's interesting because this article shows the overlap of what a non-tech thinks is AI and what is common fodder for any decent programmer. So many things get lost in buzzword to English translation, it's easy to forget that most people correlate the plastic box sitting in front of them with an intelligent Magic 8 Ball.

Re: You probably don't need AI/ML. You can make do with well written SQL scripts

#33

Post by edraferi »

philliphaydon wrote: post #11
threeseed wrote: post #4

Who on earth are these people describing ? I've never heard of anyone hiring expensive Data Scientists, spinning up Spark/H2O clusters, building a data lake, doing a database offload to S3/HDFS all for a "select from orders table where basket size is the biggest" query. AI/ML doesn't even work like this. It's simply not designed for giving 100% accurate answers to highly structured queries.

> Who on earth are these people describing ? I was asked about my thoughts on AI/ML at work, I said it didn't really apply to us. I was told "but with ML we can figure out when deliveries are happening and scale the machines before the deliveries happen based on the peak traffic times". I tried to explain that we could so all that from SQL and looking at our data. We have all the data we just need to formulate it int…

The catch it the “and looking at our data” part. ML is basically a collection of thorough ways to look at your data, understand the patterns and infer what that means for the future.

In your example, you should absolutely start by cleaning your data up and run some basic SQL aggregations and plotting volume over time. So you look that that and notice (1) volume is increasing over time, (2) some holidays bump a few days ahead but drive very low volume day of (3) weekends are higher, but the effect isn’t pronounced the whole year and (4) summer is better for you than winter except for the Christmas season. Now: it’s two days before Halloween, what’s our anticipated sales volume?

If you baked all those observations into an ARIMA model, it’s trivial to crank out a forecast with quantifiable accuracy. If you just have lines on a graph, it’s hard to pin down all the independent effects and recombine them for arbitrary scenarios.

Re: You probably don't need AI/ML. You can make do with well written SQL scripts

#34

Post by sixdimensional »

Well, we are in the peak of a wave of hype about AI/ML, maybe even just past that peak. Many fundamental technological advancements in the field of AI/ML have sort of coalesced together at the current time to form a strong feature set that can be more broadly applied by a wider audience, not just those hardcore computer scientists who invented the technology.

I've been in the thick of this previously, facing a complex rules-based engine that did most of its incredible feats in the fraud detection domain using a number of really complicated SQL queries. At the same time, I've used the results of such queries combined together with machine learning and predictive analytics, giving you the best of both worlds. Both have strengths and weaknesses.

These are tools in the toolbox, and I think the adage "try to use the best tool for the job" still applies. Sometimes, you use the tool you have and you know, and all the more power to you if you can get the job done using that tool. If you are a master of that tool (i.e. SQL in this case), you can often push its capabilities very, very far.

That said, I think the best thing to do right now is try to separate the signal from the noise regarding AI/ML and find what really works and what does not. Then find how these new tools can either complement or replace previous approaches. I think they work together quite nicely - and we see that sometimes, for example, with AI/ML tools integrated close to SQL engines.

AI/ML has a place, and so does SQL. I will say, though, that I for one don't want to be caught on the side of the discussion where I don't learn enough about what is possible with AI/ML, and then get left behind. I think many of my colleagues and professionals in the field and here on YC feel similarly.

Actually, I think even non-technical people feel the same way - the fear of being replaced by AI/ML is higher than ever.

So, keep applying SQL and get that low-hanging fruit. But make sure to learn the new stuff too, and add it to your toolbox.

Re: You probably don't need AI/ML. You can make do with well written SQL scripts

#35

Post by cosmie »

threeseed wrote: post #4

Who on earth are these people describing ? I've never heard of anyone hiring expensive Data Scientists, spinning up Spark/H2O clusters, building a data lake, doing a database offload to S3/HDFS all for a "select from orders table where basket size is the biggest" query. AI/ML doesn't even work like this. It's simply not designed for giving 100% accurate answers to highly structured queries.

Anecdotally[1] speaking, you're correct but you're missing the message. You're correct that no one should hire expensive data scientists for this. But what happens is that there's no marketing against these sorts of pragmatic best practices, so it never comes on the radar of business executives making decisions. Instead they're inundated with ML/AI/Data Science pitches and mentions everywhere. And so when they go to invest in improvements like this, they reach towards the buzzwords they know instead of the solutions they're not aware of.

What ends up happening is effectively the Data Science engagement becomes 90% data cleaning, a handful of SQL statements that should have existed beforehand but never did because the data infrastructure wasn't there, and possibly a veneer of ML/AI just to say it was used. Clients come out happy (sometimes), despite overpaying for what was a much more basic engagement than they think it was, and they go on preaching to their business exec friends the virtue of ML/AI and the cycle continues.

[1] I built up a Business Intelligence/Analytics team at my last job, and currently work for a marketing agency managing digital analytics for Fortune 50 clients. Lots of exposure to analytics in lots of varying environments, and I've seen firsthand how ML engagements get pitched and results get presented.

I also own a consultancy that's the anti-version of this phenomenon, offering digital analytics management and support services. 50% of my work involves being a knowledgeable resource for marketing and business execs to lean on to cut through the bullshit. With most of the rest being basic Google Analytics/Google Tag Manager management, CrazyEgg, and drip marketing campaigns. All of which seems like AI-level magic for clients when done correctly.

Re: You probably don't need AI/ML. You can make do with well written SQL scripts

#36

Post by nostrademons »

reilly3000 wrote: post #17

This is the opposite of AI use cases in marketing. You are declaring a specific timeframe for your message delivery. That is not how a marketer should deploy AI. I haven’t been in any pitch meetings since AI assclownery took hold so I can’t comment on how the term is being abused. What I can say is that a model that used AI would take every parameter it could about each customer and determine the optimal time to sent…

I think the main point of the article is that 99% of the value in that is in a.) sending the follow-up email at all (which just takes a cronjob) and b.) identifying which customers to send that follow-up e-mail to (which just takes a SQL query). While it's probably nice to try and predict the ideal time to send it, the gains you get from that are marginal compared to steps a & b, which many companies aren't even doing today.

Re: You probably don't need AI/ML. You can make do with well written SQL scripts

#37

Post by 1123581321 »

mygo wrote: post #6
soared wrote: post #5

Earlier quoted context omitted.

Agreed. The advertising agency I previously worked at would take the first month and only work on how data is collected and stored. After that first month they would begin addressing goals, plans, etc. About a year later is when true value would be realized because data that previously never existed could be analyzed. Few things were more enjoyable for me than getting a new client, imagining what analysis I'd like to…

how did you convince them to stick on for a year when most clients want to see some results within 2 months or they give up? according to your schedule they’d ask what I’ve been doing and if I tell them I’ve just been “collecting data” that translates to them as “I haven’t done anything in 2 months”. If I say “keep paying me and you’ll see results in a year” that translates to “I haven’t done anything in 2 months but…

Companies with budgets are willing to think in terms of quarters or years. You’ve laid out the plan for them and they’ve agreed to it. The money is allocated in an annual budget or obtained with a purchase order, so it’s not going to be suddenly reallocated unless there’s a big problem. Experienced clients wouldn’t move to another agency to do analytics “faster” because they would know they’d just have to ramp up the relationship and the systems again.

Re: You probably don't need AI/ML. You can make do with well written SQL scripts

#38

Post by cortesoft »

hahla wrote: post #14
threeseed wrote: post #4

Who on earth are these people describing ? I've never heard of anyone hiring expensive Data Scientists, spinning up Spark/H2O clusters, building a data lake, doing a database offload to S3/HDFS all for a "select from orders table where basket size is the biggest" query. AI/ML doesn't even work like this. It's simply not designed for giving 100% accurate answers to highly structured queries.

These people are describing 99% of the Fortune 500 companies who have no idea what AI means other than hiring a team of data scientists that will hopefully solve all of their problems in the name of technology.

Man, I am really curious what position you hold that you know the AI strategy for 99% of Fortune 500 companies. Those same Fortune 500 companies would pay you a lot of money for this level of insight into their competitors.

Re: You probably don't need AI/ML. You can make do with well written SQL scripts

#40

Post by gnicholas »

My startup was approached by a corporate VC that wanted to make a strategic investment. Based on the attendee list from our meeting, which included very high up folks from the company, I felt good going in. They expressed interest in our technology that makes reading on screen easier [1], but they were surprised to learn that we didn't use machine learning to accomplish this.

I indicated that it was actually quite effective without ML, and that it was easier to explain to users this way. They kept prodding around on the ML stuff, and how we might be able to use ML to accomplish roughly the same thing.

A week later they said that they were no longer interested because, although they liked what our tech was able to accomplish, it didn't fit with their investment thesis — which was all about ML.

My wife asked me why I didn't just make some stuff up and say we could do v2 using ML. Perhaps she was right.

1: http://www.beelinereader.com/individual

update: in response to feedback below, I edited the link to point to a page with relevant content instead of our generic landing page. Lesson learned!

Post reply on HN