Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ways Data Projects Fail

martingoodson.com

Post reply on HN
Print view View original

61–70 of 79 posts

Re: Ways Data Projects Fail

#61

Post by IanCal »

eanzenberg wrote: post #27
jackschultz wrote: post #2

Good to see that data cleaning was #1 on that list. Whenever I do work on a side project, it takes way way more time to get and structure the data than it does running the algorithms. Granted, that's because I have to go out and get the data in the first place, and then make sure it's useable and in the correct format. Like the recent project I'm doing trying to classify country music songs based on their topic on th…

>> I've been looking for jobs recently, and I've seen only one job posting that mentions data cleaning as a necessity, whereas the rest only talk about data science and algorithm knowledge, or overall ETL design on the data engineering side. Seems like data set knowledge should be emphasized more. Actual data cleaning, usually in an automated sense, is more 'data engineering' than 'data science' or applied statistics…

I'd hesitate to call a lot of the work I do cleaning data "engineering".

I think perhaps the problem here is the term science covers a lot of disciplines.

I propose harder stats be data theoretical physics, with data biology and similar referring to cases with harder messy real world complications. I'm sure we can come up with a full spectrum.

Re: Ways Data Projects Fail

#62

Post by dasboth »

makmanalp wrote: post #3
jackschultz wrote: post #2

Good to see that data cleaning was #1 on that list. Whenever I do work on a side project, it takes way way more time to get and structure the data than it does running the algorithms. Granted, that's because I have to go out and get the data in the first place, and then make sure it's useable and in the correct format. Like the recent project I'm doing trying to classify country music songs based on their topic on th…

Yes! Came here to write this too. I've been spending the last few years thinking, inquiring and talking about this, and I think there definitely is a field emerging here. There are already some companies trying to think about this stuff, and classes give it lip service but I don't think we've even seen the tip of the iceberg. It's interesting to talk to different people about data quality and what they think it means…

> classes give it lip service

I agree with this. It's fine to teach machine learning using the iris dataset, but there is rarely, if ever, a section dedicated to "real" problems. It was a shock to me just how high a percentage of time is spent cleaning data. It is a fundamental skill that is not only underestimated but "undertaught".

Re: Ways Data Projects Fail

#63

Post by dasboth »

eanzenberg wrote: post #27
jackschultz wrote: post #2

Good to see that data cleaning was #1 on that list. Whenever I do work on a side project, it takes way way more time to get and structure the data than it does running the algorithms. Granted, that's because I have to go out and get the data in the first place, and then make sure it's useable and in the correct format. Like the recent project I'm doing trying to classify country music songs based on their topic on th…

>> I've been looking for jobs recently, and I've seen only one job posting that mentions data cleaning as a necessity, whereas the rest only talk about data science and algorithm knowledge, or overall ETL design on the data engineering side. Seems like data set knowledge should be emphasized more. Actual data cleaning, usually in an automated sense, is more 'data engineering' than 'data science' or applied statistics…

That also depends on the team. If your company has dedicated data engineers, great! Otherwise you're probably stuck with it.

Re: Ways Data Projects Fail

#64

Post by dasboth »

jupiter90000 wrote: post #46
Declanomous wrote: post #12

>Your Data Scientists are about to quit. This is me. I work for a non-profit that is stuck in the stone age--not for lack of money, mind you, but because the IT Director is an incompetent megalomaniac who views "security" as a reasonable justification to refuse any and all requests, and treats everyone like an enemy. I haven't been allowed to use Python or R. In fact, the only programming language I have access to is…

That sounds horrible. Ever thought of writing a VBA for data science book? ;-)

The world doesn't need that! It shouldn't be encouraged!

Re: Ways Data Projects Fail

#65

Post by blahi »

TeMPOraL wrote: post #19
searine wrote: post #6

Earlier quoted context omitted.

Hypothesis driven data science is over the head of too many people.

That's why it's called "data mining" - you keep digging until you find something. And with lots of data, you can always find something if you look hard enough - which leads to things like http://tylervigen.com/discover .

Data drudging is what it is called.

Re: Ways Data Projects Fail

#66

Post by hssay »

martingoodson wrote: post #10

Author here - in case of any criticisms or comments.

Thanks for the excellent article.

One more interesting thing I have observed in data projects failing: organizations culture around data and the gap between data science team and engineers. Say, you have 2 top notch data scientist who know enough (stats, markov chains, algorithms and so on..). But let us say an average engineer in the organization doesn't know even a bit about A/B testing or difference between building a machine learning model Vs. obtaining predictions from already built model. Then no matter how good your so called data scientist are, the end result in terms of product or solution delivery is always sub-optimal. If the engineers and data science teams can't speak a common language, the result is always disastrous. Note that the gap is specifically about understanding data analysis as a domain.

The efforts to narrow down this gap must be driven by the lead data science member or CTO. Something like 'data bootcamp' mandatory for every new joinee can help. I had read about Facebook having such a bootcamp mandatory.

Re: Ways Data Projects Fail

#67

Post by mrweasel »

jackschultz wrote: post #2

Good to see that data cleaning was #1 on that list. Whenever I do work on a side project, it takes way way more time to get and structure the data than it does running the algorithms. Granted, that's because I have to go out and get the data in the first place, and then make sure it's useable and in the correct format. Like the recent project I'm doing trying to classify country music songs based on their topic on th…

I'm currently preparing a lecture on the topic of logging for my students. Part of the lecture is of cause how to use various logging frameworks, but the main part is what to log and how to structure logs.

Basically we're trying to get them to pre-emptively do data cleaning, so their logs will actually be useful for potential future data projects.

Re: Ways Data Projects Fail

#68

Post by vram22 »

gerhardi wrote: post #59
jackschultz wrote: post #2

Good to see that data cleaning was #1 on that list. Whenever I do work on a side project, it takes way way more time to get and structure the data than it does running the algorithms. Granted, that's because I have to go out and get the data in the first place, and then make sure it's useable and in the correct format. Like the recent project I'm doing trying to classify country music songs based on their topic on th…

Hundred times this. You see Qlik/Cognos Analytics/PowerBI/Alteryx/whatever sales guys making demos that make executives drool over the seeming easiness and wow-factor these tools are capable of producing. When the time comes to plug those over your production operative systems, CRM, whatelse, there comes "the now wait a minute" moment especially if your systems and their data models happen to be even slightly on the…

MDM is Master Data Management? (from a quick google). Hadn't heard of that specific term before. I'm interested in data projects.

Re: Ways Data Projects Fail

#69

Post by gerhardi »

vram22 wrote: post #68
gerhardi wrote: post #59

Earlier quoted context omitted.

Hundred times this. You see Qlik/Cognos Analytics/PowerBI/Alteryx/whatever sales guys making demos that make executives drool over the seeming easiness and wow-factor these tools are capable of producing. When the time comes to plug those over your production operative systems, CRM, whatelse, there comes "the now wait a minute" moment especially if your systems and their data models happen to be even slightly on the…

MDM is Master Data Management? (from a quick google). Hadn't heard of that specific term before. I'm interested in data projects.

Yes correct. In a way it's a combination of philosophy, agreed practices and techical solutions. I think this one is a good introduction: The What, Why, and How of Master Data Management https://msdn.microsoft.com/en-us/library/bb190163.aspx

Re: Ways Data Projects Fail

#70

Post by Declanomous »

maxxxxx wrote: post #40
Declanomous wrote: post #12

>Your Data Scientists are about to quit. This is me. I work for a non-profit that is stuck in the stone age--not for lack of money, mind you, but because the IT Director is an incompetent megalomaniac who views "security" as a reasonable justification to refuse any and all requests, and treats everyone like an enemy. I haven't been allowed to use Python or R. In fact, the only programming language I have access to is…

To be honest your IT director sounds like an idiot but the real problem are YOUR managers. They should have protected you and told the IT director to f...k off a long time ago.

I've had the same thought. I actually met with my boss and HR about this, and they said there was nothing they could do. The issue is the company structure. I work for a 501c3 affiliated to another organization. The parent company provides all of our admin functions like HR, IT, etc. We share a board, and the director of each department reports directly to the board. The IT director is really well liked by the board because he never spends any money.

The average age of our board is somewhere north of 80 (not kidding), and they don't understand what IT is. True story--our parent organization didn't have a website until 2003, because the IT director thought websites were a fad. He was forced to buy the domain after someone else bought it and used it to post stuff the board found unappealing.

The only way to describe the entire situation is Kafkaesque.

Post reply on HN