Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Be good-argument-driven, not data-driven

twitchard.github.io

Post reply on HN
Print view View original

71–80 of 168 posts

Re: Be good-argument-driven, not data-driven

#71

Post by selykg »

packetlost wrote: post #69
shubb wrote: post #35

The related problem that I see actually more often is the "you don't have big data" problem. You know, in data science, you see people spending hours writing pandas scripts that replicate a few clicks in excel for a one of analysis. You see datasets of a few gigabytes being processed with spark when SQL would be fine. You see ML techniques being thrown at questions that could be answered simply and reliably with basi…

> You know, in data science, you see people spending hours writing pandas scripts that replicate a few clicks in excel for a one of analysis I mean, having an Excel doc at all usually implies hour(s) of work formatting the data in structured manner. Sometimes collective decades of work depending on how much heavy lifting your 15GB .xlsx is doing.

This is why I've adopted R and Python for the data work I do. I have a bunch of exported data (CSV files) that I use. Manipulating the structure and format is 90% of the work. I wrote the scripts once, now I can reuse that for everything instead of playing games getting those CSV files (dates in particular) to play nicely.

Even a one off analysis is actually FASTER in Pandas because I've done the work of farting around with the formatting. Now I can just write the necessary analysis code, rather than deal with the formatting.

That said, my data analytics work is seriously small potatoes compared to many. But I can write a quick pivot table using Dplyr faster than I can do it in Excel.

Re: Be good-argument-driven, not data-driven

#72

Post by k__ »

shubb wrote: post #35

The related problem that I see actually more often is the "you don't have big data" problem. You know, in data science, you see people spending hours writing pandas scripts that replicate a few clicks in excel for a one of analysis. You see datasets of a few gigabytes being processed with spark when SQL would be fine. You see ML techniques being thrown at questions that could be answered simply and reliably with basi…

Would you say the big data threshold moves every year?

That would explain why people think a <1TB is big data.

Re: Be good-argument-driven, not data-driven

#73

Post by culturestate »

safety1st wrote: post #63
viridian wrote: post #46

This entire discussion makes a good case for why the general populace would benefit from being taught the basics of philosophy. In this case the topic of value is the often fraught relationship between empiricism and rationalism , and the impacts each have on the scientific process, research, education, and how we go about understanding the world. To operate with one with a complete absence of the other is to expose…

I would start by simply putting everyone through a course in deductive reasoning at the earliest age possible: https://en.wikipedia.org/wiki/Deductive_reasoning From there you can go into the whole spectrum of critical thinking approaches, and then on to what's basically the liberal arts e.g. philosophy, social sciences etc. as you desire. But the value you get from all of those things depends heavily on the framewor…

> I would start by simply putting everyone through a course in deductive reasoning at the earliest age possible

I was taught the explicit premise of deductive vs. inductive reasoning as part of our unit on the scientific method in, I think, fourth or fifth grade. I always assumed this was a standard curriculum module.

Re: Be good-argument-driven, not data-driven

#74

Post by jkingsbery »

germinalphrase wrote: post #68
bell-cot wrote: post #41

Earlier quoted context omitted.

Unfortunately, your reality-driven approach has ~zero emotional appeal for most managers, exec's, and alpha-data-scientist wanna-be's.

Why? Inadequately “technical”?

I think there are lots of reasons why.

One possible reason: no one whose job it is to write Python scripts was ever promoted for making an Excel spreadsheet when that is the simpler and more practical approach. And no manager of people who write Python scripts is going to be able to use that Excel spreadsheet to sell "I need more responsibility and head count." People tend to follow incentives, rather than focusing on making wise decisions.

Re: Be good-argument-driven, not data-driven

#75

Post by gampleman »

shubb wrote: post #35

The related problem that I see actually more often is the "you don't have big data" problem. You know, in data science, you see people spending hours writing pandas scripts that replicate a few clicks in excel for a one of analysis. You see datasets of a few gigabytes being processed with spark when SQL would be fine. You see ML techniques being thrown at questions that could be answered simply and reliably with basi…

I say this about every other day at work (we even have only internal users so it's part of their job to talk to us). So far impact: zero....

Re: Be good-argument-driven, not data-driven

#76

Post by bogdanoff_2 »

ekianjo wrote: post #42
shubb wrote: post #35

The related problem that I see actually more often is the "you don't have big data" problem. You know, in data science, you see people spending hours writing pandas scripts that replicate a few clicks in excel for a one of analysis. You see datasets of a few gigabytes being processed with spark when SQL would be fine. You see ML techniques being thrown at questions that could be answered simply and reliably with basi…

Talking to people is not going to help you either. You end up getting a lot of noise and making sense of what you hear is difficult. When you keep probing you will get to hear stuff thats not really critical and just often made up because you ask too many questions. Classical trap of market research.

Talking to customers might uncover some things you haven't even thought about.

Re: Be good-argument-driven, not data-driven

#78

Post by shubb »

stevofolife wrote: post #67
shubb wrote: post #35

The related problem that I see actually more often is the "you don't have big data" problem. You know, in data science, you see people spending hours writing pandas scripts that replicate a few clicks in excel for a one of analysis. You see datasets of a few gigabytes being processed with spark when SQL would be fine. You see ML techniques being thrown at questions that could be answered simply and reliably with basi…

Why not both…

Maybe what I wrote comes off a bit one sided - I'm really urging people to do what actually makes sense in their specific context - which can be both!

Re: Be good-argument-driven, not data-driven

#80

Post by dspillett »

Cthulhu_ wrote: post #61
ekianjo wrote: post #42

Earlier quoted context omitted.

Talking to people is not going to help you either. You end up getting a lot of noise and making sense of what you hear is difficult. When you keep probing you will get to hear stuff thats not really critical and just often made up because you ask too many questions. Classical trap of market research.

This is true; what customers SAY they want doesn't necessarily corellate with what they will actually use or pay for. I mean I worked on an app where in one part, the end user could upload CSV files to be used. What they SAID they wanted was basically a full data management system and RESTful API to enforce constraints, data validation, record retrieval and updating, etc. What they probably wanted was an excel sheet.…

> what customers SAY they want doesn't necessarily corellate with what they will actually use

A key cause of this in many cases is that the stake-holders you talk to do not work closely with the end users of the system. Talking to the right people can help a lot, though unfortunately as a 3rd party this is not usually anywhere near your realm of control.

The other issue is them knowing what they have and wish to store, but not knowing what outputs are going to be needed down the line. That is harder to fix, but having some good industry knowledge within your company can be a great help on such matters – you can then sometimes preempt client needs if the people holding that knowledge are keeping an active eye on changes (for instance new/planned regulations that might be coming into force in X weeks/months/years).

Post reply on HN