Datasets You've Likely Never Seen • Hacker News Board

Re: Datasets You've Likely Never Seen

#11

Post by tfgg » Thu, Jul 02, 2015, 8:55 AM UTC

roel_v wrote: post #10

ObviousScience wrote: post #9

Earlier quoted context omitted.

I'm not sure I understand your last sentence; could you elaborate?

What I mean is, that many people are still stuck at the 'we need more data' stage, or at least at the 'better data collection/verification'. And that the focus of much analysis and modeling is less on actionable advice, but more... well how shall I put it, 'dissecting' data, without having a clear way in mind how that dissection will lead to insights that are relevant for the stakeholder. I should probably mention th…

Couldn't it still be the case, though, that we have too much data, but it's also the wrong sort for actionable insights? As a scientist I find the most actionable data are often in the smallest, custom-made datasets, driven by some question, not trawling through masses of data collected without a goal, hoping that they'll have collected the right thing.

Re: Datasets You've Likely Never Seen

#12

Post by roel_v » Thu, Jul 02, 2015, 9:49 AM UTC

tfgg wrote: post #11

roel_v wrote: post #10

Earlier quoted context omitted.

What I mean is, that many people are still stuck at the 'we need more data' stage, or at least at the 'better data collection/verification'. And that the focus of much analysis and modeling is less on actionable advice, but more... well how shall I put it, 'dissecting' data, without having a clear way in mind how that dissection will lead to insights that are relevant for the stakeholder. I should probably mention th…

Couldn't it still be the case, though, that we have too much data, but it's also the wrong sort for actionable insights? As a scientist I find the most actionable data are often in the smallest, custom-made datasets, driven by some question, not trawling through masses of data collected without a goal, hoping that they'll have collected the right thing.

Sure, could very well be, and what is a perfect fit in one situation, might be unusable in another situation that at the surface looks like it's almost the same. It would be silly for me to claim that all data we need is already being collected or something like that. But that's not at odds with my abstract point that the realization that data is usually no longer the problem, but the lack of knowing what to do with it hasn't sunk in for most people. (This makes it sound like I think of myself as someone who has seen The Light and 'those others' are chumps, which would obviously be delusional of me, and I don't mean it that way)

I guess what I'm failing to articulate here is the shift that has snuck up on us over the last 10 or so years. My bitching about the quality of datasets today is about increasingly marginal issues (at the macro scale of course, there are still crap individual datasets, obviously); whereas 15 years ago, I didn't even have datasets to bitch about.

Re: Datasets You've Likely Never Seen

#13

Post by jsuar » Thu, Jul 02, 2015, 1:42 PM UTC

roel_v wrote: post #10

ObviousScience wrote: post #9

Earlier quoted context omitted.

I'm not sure I understand your last sentence; could you elaborate?

What I mean is, that many people are still stuck at the 'we need more data' stage, or at least at the 'better data collection/verification'. And that the focus of much analysis and modeling is less on actionable advice, but more... well how shall I put it, 'dissecting' data, without having a clear way in mind how that dissection will lead to insights that are relevant for the stakeholder. I should probably mention th…

This has been my experience while taking masters courses, but I assumed it was because it's important to learn about all the available tools and techniques. Has there been no research on methodologies that aid in discovering insights?

It reminds me of the difficulties in teaching someone how to prove statements in math. The basic approach is to learn as many techniques as possible (tools in your tool bag), review existing proofs, and practice. You really can't teach someone how to find the connections and insights that lead to a proof. I was once told that was the art/creativity in proofs.

Re: Datasets You've Likely Never Seen

#14

Post by MrZongle2 » Thu, Jul 02, 2015, 3:52 PM UTC

waterlesscloud wrote: post #2

I now have a spreadsheet of Spanish Silver Production 1720-1800 for no other reason than it was available to me. Well, and a long lingering interest in Sid Meier's Pirates and Neal Stephenson's Baroque Cycle.

Well then, here's timing for you... Jimmy Maher, who has an excellent blog that chronicles the evolution of the game industry, posted an article about Pirates! today: http://www.filfre.net/2015/07/pirates/