Live data from Hacker News

Ways Data Projects Fail

martingoodson.com

41–50 of 79 posts

Re: Ways Data Projects Fail

#41

Earlier quoted context omitted.

Yes! Came here to write this too. I've been spending the last few years thinking, inquiring and talking about this, and I think there definitely is a field emerging here. There are already some companies trying to think about this stuff, and classes give it lip service but I don't think we've even seen the tip of the iceberg. It's interesting to talk to different people about data quality and what they think it means…

Data governance and master data management spring to mind as existing ideas along these lines.

Aye, but the locked-down nature of data governance actually fights against using it for insights. I've seen a situation where a small two-way frequency table was requested by Team 1, and it took six _months_ because Team 2 had access to data but no access to metadata, Team 3 had metadata and could help design a query but didn't understand the technical details or run the query, Team 4 had to approve the process, and Team 5 had to review Team 3's query before it could run. In the mean time Team 2 was reorged and Team 1, the original requestors, found upstream sources.

Data in a regulatory regime can be excruciatingly difficult, and lend itself to "gut instinct" being used because fear of risk and regulation lock things down too tight to be useful.

Re: Ways Data Projects Fail

#42

Author here - in case of any criticisms or comments.

A lot of data scientists these days (me included) are former academics with backgrounds in numerical simulation in fields like chemistry, physics, mechanical engineering etc. They live and breath numerical linear algebra and are comfortable reading advanced theoretical books or papers. It's easy for them to pick up the basics needed to pass interviews and find a data science job. How would they go about adding some r…

I wouldn't expect the majority of data science jobs to be particularly focused on the math behind the algorithms. Rudimentary understanding of probability and how to translate the jargon into your academic background's jargon is more important than deep understanding for these jobs. Passing the interviews for these jobs is one thing. Unless you're specifically looking for jobs that focus on generating new modeling techniques or algorithms for computational statistics, expect to be far removed from even basic linear algebra in actual practice. Source: me. I fall in your described bucket and have worked in data science/machine learning jobs in both contexts (new modeling techniques/stats versus application of off-the-shelf tools).

Re: Ways Data Projects Fail

#43

Good to see that data cleaning was #1 on that list. Whenever I do work on a side project, it takes way way more time to get and structure the data than it does running the algorithms. Granted, that's because I have to go out and get the data in the first place, and then make sure it's useable and in the correct format. Like the recent project I'm doing trying to classify country music songs based on their topic on th…

I agree - I have a project that has similar problems (cataloging standalone lectures - https://findlectures.com). The biggest advantage of it being a side project is there's no pressure to get the data cleaning done, but in a work environment with time pressure this type of project is a huge pain.

Re: Ways Data Projects Fail

#45

>Your Data Scientists are about to quit. This is me. I work for a non-profit that is stuck in the stone age--not for lack of money, mind you, but because the IT Director is an incompetent megalomaniac who views "security" as a reasonable justification to refuse any and all requests, and treats everyone like an enemy. I haven't been allowed to use Python or R. In fact, the only programming language I have access to is…

Wow that's incredible. I'm sure you know this by now, but I don't think there's any fixing that situation. Good luck!

Re: Ways Data Projects Fail

#46

>Your Data Scientists are about to quit. This is me. I work for a non-profit that is stuck in the stone age--not for lack of money, mind you, but because the IT Director is an incompetent megalomaniac who views "security" as a reasonable justification to refuse any and all requests, and treats everyone like an enemy. I haven't been allowed to use Python or R. In fact, the only programming language I have access to is…

That sounds horrible. Ever thought of writing a VBA for data science book? ;-)

Re: Ways Data Projects Fail

#47

Author here - in case of any criticisms or comments.

I enjoyed the conciseness of the article, but I don't have a big stake in this field so I can't comment on the content itself. I wanted to make a small formatting note: having key parts of the article only as images (not reproduced or described in the text) is bad for accessibility, search engines, etc. an article should still make sense if you take out all the images

Re: Ways Data Projects Fail

#48
post #41

Earlier quoted context omitted.

Data governance and master data management spring to mind as existing ideas along these lines.

Aye, but the locked-down nature of data governance actually fights against using it for insights. I've seen a situation where a small two-way frequency table was requested by Team 1, and it took six _months_ because Team 2 had access to data but no access to metadata, Team 3 had metadata and could help design a query but didn't understand the technical details or run the query, Team 4 had to approve the process, and…

I couldn't agree more. Bureaucracy creates silos and stymies sharing and agility. Balance is needed.

Re: Ways Data Projects Fail

#50
An honest question. If I can make really good money helping businesses make sense of out "smallish" data in Excel, why would I subject myself to the miasma that is "data science"? Will I be able to charge lawyer-like rates?
Post reply on HN