Live data from Hacker News

Ways Data Projects Fail

martingoodson.com

11–20 of 79 posts

Re: Ways Data Projects Fail

#11

Good to see that data cleaning was #1 on that list. Whenever I do work on a side project, it takes way way more time to get and structure the data than it does running the algorithms. Granted, that's because I have to go out and get the data in the first place, and then make sure it's useable and in the correct format. Like the recent project I'm doing trying to classify country music songs based on their topic on th…

Any specific resources you'd recommend on data cleaning, verification, etcetera? I've just started reading this: https://www.amazon.com/Accuracy-Economic-Observations-Oskar-... . I've seen a few other books on the subject which I'm planning to get into, but I'd be interested if anyone has specific recommendations.

Do you use R and the "hadleyverse"? (Or "tidyverse" I think as he prefers?)

I'm a programmer by trade but I use R because the people who actually work with data use it, and they write good tools for it... I think there is some confusion in the programming world about this. Programmers work with data, but they don't do it nearly as much as "professionals".

Tidy data is a good intro if you're not familiar with it:

http://vita.had.co.nz/papers/tidy-data.html

And I would recommend going through other publications by Wickam, all on his site -- they are quite readable.

Re: Ways Data Projects Fail

#12
>Your Data Scientists are about to quit.

This is me. I work for a non-profit that is stuck in the stone age--not for lack of money, mind you, but because the IT Director is an incompetent megalomaniac who views "security" as a reasonable justification to refuse any and all requests, and treats everyone like an enemy.

I haven't been allowed to use Python or R. In fact, the only programming language I have access to is VBA (for applications, not the stand-alone variant). Of course that's a huge mess because the IT director disables macros once a month, generally right after another crypto attack makes the news. Thankfully, he didn't even realize that it was possible to use VBA from inside any office application until after I had already used it to create several Access applications which made the jobs of the most important people in the organization easier. So when he breaks VBA every director in the organization yells at him and the functionality is restored nearly instantly.

Of course he could restrict the applications to run only signed macros, but he won't give me permission to sign things because he is (literally) afraid I might hack something.

On top of that, my computer is a Core 2 Duo from 2007 or so with 4 gb of ram. He bought over 100 of them used from a computer recycler about 2 years ago. For the first three months at this job I had a Pentium D, which literally couldn't run Excel and Firefox at the same time. I'm not allowed to get a better computer, because the employee handbook states that every computer needs to be the same for "security" reasons. If my director used our budget to purchase a computer I wouldn't be granted access to any of the databases containing our data because of "HIPAA compliance." (For the record, we don't have any medical data whatsoever. We only have names, addresses, and donation amounts. We don't even know the birthdays of our constituents.)

The worst part is that we randomly started losing data after all of our network drives were moved offsite at one point to provide "redundancy." I created several tickets about this issue, and each time I was told that it couldn't have possibly happened, and there was no record of the file ever existing. I created a script that created a log file each hour with a list of files and their attributes from each directory to try record proof of this happening. After I recorded about a week of files disappearing randomly overnight, he reported me to HR for hacking.

Once I proved nothing I did was wrong he amended the "IT security" section of the employee handbook. Several of these measures were impossible to follow because of restrictions he had placed on the computers/network. I brought this up with HR, and they removed these measures from the handbook. Once this happened, he sent an email to me cc'ing my boss and HR accusing me of trying to frame him by deleting files. I don't know how that accusation even made sense, because the files would still have to show up in transaction logs.

Despite all this, I KNOW my director and HR aren't going to believe me when I tell them I'm quitting because our IT director is an incompetent tyrant. From their perspective, IT issues are something that can be solved by compromise, just like everything else. So IT has to let me use VBA, and that should be enough.

Anyways, long story short, anybody hiring in Chicago?

Re: Ways Data Projects Fail

#13
post #5

The big one that's missing: There is nothing you can conclude from your data. It's clean, it makes its way properly to the analyst, and yet, there's just nothing there...

A great example of this was reported to me from a head of data science at a (UK) national newspaper. The business set a task of predicting subscriber numbers using online user behaviour from the previous month as features. But it turned out that most subscribers converted within three days of first hitting the site. The previous month of data was almost completely worthless. Like you say, the data just didn't contain the information that it 'should' have.

Re: Ways Data Projects Fail

#14

>Your Data Scientists are about to quit. This is me. I work for a non-profit that is stuck in the stone age--not for lack of money, mind you, but because the IT Director is an incompetent megalomaniac who views "security" as a reasonable justification to refuse any and all requests, and treats everyone like an enemy. I haven't been allowed to use Python or R. In fact, the only programming language I have access to is…

Did you not see the Who is Hiring thread yesterday...?

Re: Ways Data Projects Fail

#16

>Your Data Scientists are about to quit. This is me. I work for a non-profit that is stuck in the stone age--not for lack of money, mind you, but because the IT Director is an incompetent megalomaniac who views "security" as a reasonable justification to refuse any and all requests, and treats everyone like an enemy. I haven't been allowed to use Python or R. In fact, the only programming language I have access to is…

Did you not see the Who is Hiring thread yesterday...?

I did. The question at the end is a rhetorical device intended to provide a bit of humor while reinforcing the author's point.

Re: Ways Data Projects Fail

#17

Author here - in case of any criticisms or comments.

A lot of data scientists these days (me included) are former academics with backgrounds in numerical simulation in fields like chemistry, physics, mechanical engineering etc.

They live and breath numerical linear algebra and are comfortable reading advanced theoretical books or papers.

It's easy for them to pick up the basics needed to pass interviews and find a data science job. How would they go about adding some rigor to their understanding of ML and statistics?

Re: Ways Data Projects Fail

#18

Good to see that data cleaning was #1 on that list. Whenever I do work on a side project, it takes way way more time to get and structure the data than it does running the algorithms. Granted, that's because I have to go out and get the data in the first place, and then make sure it's useable and in the correct format. Like the recent project I'm doing trying to classify country music songs based on their topic on th…

Any specific resources you'd recommend on data cleaning, verification, etcetera? I've just started reading this: https://www.amazon.com/Accuracy-Economic-Observations-Oskar-... . I've seen a few other books on the subject which I'm planning to get into, but I'd be interested if anyone has specific recommendations.

Well this is certainly a good start: https://github.com/Quartz/bad-data-guide

Re: Ways Data Projects Fail

#19
post #6
post #5

The big one that's missing: There is nothing you can conclude from your data. It's clean, it makes its way properly to the analyst, and yet, there's just nothing there...

Hypothesis driven data science is over the head of too many people.

That's why it's called "data mining" - you keep digging until you find something. And with lots of data, you can always find something if you look hard enough - which leads to things like http://tylervigen.com/discover.

Re: Ways Data Projects Fail

#20
In spite of assurances from business process owners that the underlying data sources are clean ... they almost certainly are not.

Multiple legacy systems with no consistent cross reference to unambiguously identify the same customer. Assured that systems have been gone through and all the names made consistent. Consistency for a human is not consistency for a computer. "Commers Ltd" is not the same as "Commers Ltd." And, isn't it lovely when a salesperson decides to add a location to a customer name. Now we have "Commers Ltd Dallas" as a unique customer. Business process discipline is often lacking and will mess you up.

Subscription data sources that change their schema with no notification to paying customers. And, when you are scraping data from websites you need to constantly be checking that your scrapers are still working properly. Source websites change regularly.

Crazy processes like entering a negative invoice to indicate a refund to customers but forgetting to zero out the cost of goods related to the invoice. We may have refunded the money but we didn't do the work twice. Arggh! Errors abound.

Post reply on HN