31–40 of 79 posts
The big one that's missing: There is nothing you can conclude from your data. It's clean, it makes its way properly to the analyst, and yet, there's just nothing there...
If done properly that's still a useful result. "None of the variables were predictive of X."
(The chance for which isn't that big when done "properly".)
Author here - in case of any criticisms or comments.
Also you misspelled "breathe". As in "live and breathe".
Author here - in case of any criticisms or comments.
I think this is great, but point #4 is way off the mark. It's pretentious and it's gatekeeping. It's not true that a strong analytics leader as you've described is incapable of understanding selection bias, measurement bias, Simpson's paradox, and statistical significance. I would posit that a great analytics leader needs to understand all of those. It's not true that only a "data scientist" with a bunch of fancy pro…
Absolutely agree that data cleaning should be at the top -- how someone prioritizes data cleaning is for me, the main litmus test to how effective they are at real-world data problems. I also agree with how the author summarizes the issue, but he also runs into the same issue I have: data cleaning is such a broad term that it obscures how difficult and important of a problem it is. For example, some people think data…
I've always disliked the term data cleaning for the reasons you mention - it doesn't tell me anything about what is meant by "cleaning".
>Your Data Scientists are about to quit. This is me. I work for a non-profit that is stuck in the stone age--not for lack of money, mind you, but because the IT Director is an incompetent megalomaniac who views "security" as a reasonable justification to refuse any and all requests, and treats everyone like an enemy. I haven't been allowed to use Python or R. In fact, the only programming language I have access to is…
always looking to meet smart people!
Earlier quoted context omitted.
If done properly that's still a useful result. "None of the variables were predictive of X."
And it's much less useful cousin hiding in the shadows "The variables are actually predictive of X, but you don't have enough data that it shows". (The chance for which isn't that big when done "properly".)
Earlier quoted context omitted.
Any specific resources you'd recommend on data cleaning, verification, etcetera? I've just started reading this: https://www.amazon.com/Accuracy-Economic-Observations-Oskar-... . I've seen a few other books on the subject which I'm planning to get into, but I'd be interested if anyone has specific recommendations.
Do you use R and the "hadleyverse"? (Or "tidyverse" I think as he prefers?) I'm a programmer by trade but I use R because the people who actually work with data use it, and they write good tools for it... I think there is some confusion in the programming world about this. Programmers work with data, but they don't do it nearly as much as "professionals". Tidy data is a good intro if you're not familiar with it: http…
>Your Data Scientists are about to quit. This is me. I work for a non-profit that is stuck in the stone age--not for lack of money, mind you, but because the IT Director is an incompetent megalomaniac who views "security" as a reasonable justification to refuse any and all requests, and treats everyone like an enemy. I haven't been allowed to use Python or R. In fact, the only programming language I have access to is…