Be good-argument-driven, not data-driven

Post by didgetmaster » Thu, Sep 01, 2022, 7:04 PM UTC

majormajor wrote: post #126

packetlost wrote: post #69

Earlier quoted context omitted.

> You know, in data science, you see people spending hours writing pandas scripts that replicate a few clicks in excel for a one of analysis I mean, having an Excel doc at all usually implies hour(s) of work formatting the data in structured manner. Sometimes collective decades of work depending on how much heavy lifting your 15GB .xlsx is doing.

Often that work exists regardless of if a table of processed data that engineering formatted and schema-fied is dumped out to Excel or queried over SQL into Pandas... I've seen this myself: the person who "naively" downloads that table and plays around in excel finds interesting things that the person who was using Pandas hadn't, because the code to manipulate columns and do certain types of calcs is actually more ti…

Post by didgetmaster » Thu, Sep 01, 2022, 7:12 PM UTC

k__ wrote: post #72

shubb wrote: post #35

The related problem that I see actually more often is the "you don't have big data" problem. You know, in data science, you see people spending hours writing pandas scripts that replicate a few clicks in excel for a one of analysis. You see datasets of a few gigabytes being processed with spark when SQL would be fine. You see ML techniques being thrown at questions that could be answered simply and reliably with basi…

Would you say the big data threshold moves every year? That would explain why people think a <1TB is big data.

Post by blooalien » Thu, Sep 01, 2022, 7:21 PM UTC

kylereeve wrote: post #127

bell-cot wrote: post #101

Earlier quoted context omitted.

Needing to CYA also has pretty low emotional appeal for managers, exec's, and alpha-data-scientist wanna-be's. (Until it's just about too late, obviously.) And recall Mark Twain's old quip about lies & statistics. The more & bigger data that the folks who control the data & analysis have, the easier it is to make sure that those meet their own emotional & political needs.

Wasn't that Will Rogers?

Post by btilly » Thu, Sep 01, 2022, 7:48 PM UTC

tlarkworthy wrote: post #82

ekianjo wrote: post #42

Earlier quoted context omitted.

Talking to people is not going to help you either. You end up getting a lot of noise and making sense of what you hear is difficult. When you keep probing you will get to hear stuff thats not really critical and just often made up because you ask too many questions. Classical trap of market research.

ycombinator startup school disagrees and says it's one of the two CRITICAL things founders must have a hand in. Of course you need to interpret it but its incredibly important and I do not think you really know what you are talking about. https://www.ycombinator.com/library/6g-how-to-talk-to-users Almost all the major fails I have seen in my career have been some derivative of not understanding your users.

Post by int_19h » Thu, Sep 01, 2022, 8:21 PM UTC

klenwell wrote: post #117

kqr wrote: post #4

While I agree completely with the premise of this article, on the other hand I'm weighing the relatively robust findings by Meehl et al. They find, time and time again, in all sorts of fields, that extremely parsimonious models like equal-weighted linear regression of one or two predictors outperform expert judgment[1]. One would think this is cognitively dissonant enough, but it gets worse: This article, with the th…

> They find, time and time again, in all sorts of fields, that extremely parsimonious models like equal-weighted linear regression of one or two predictors outperform expert judgment. I came across this in Thinking Fast and Slow. Kahneman was a big fan of Meehl and restates the point: The important conclusion from this research is that an algorithm that is constructed on the back of an envelope is often good enough t…