Data Is Never Raw • Hacker News Board

Re: Data Is Never Raw

#11

Post by teej » Thu, Jan 10, 2019, 5:41 PM UTC

eximius wrote: post #5

LanceH wrote: post #4

Earlier quoted context omitted.

Have 100,000 people enter there address with a text field for the state and get back to me with how many states there are.

50 states, probably. ;) Edit: assuming a very US-centric dataset...

Plus Washington DC, the US territories, and US military bases abroad.

Re: Data Is Never Raw

#12

Post by rhacker » Thu, Jan 10, 2019, 5:47 PM UTC

If someone hired me a data scientist and I said: "Can you help me process this raw data into something that we can work with in this tensorflow model?" And they reply with:

"Don't call it Raw Data"

I would ask my manager to get me someone else.

Re: Data Is Never Raw

#13

Post by rhacker » Thu, Jan 10, 2019, 9:13 PM UTC

icedchai wrote: post #6

eximius wrote: post #5

Earlier quoted context omitted.

50 states, probably. ;) Edit: assuming a very US-centric dataset...

50 states, represented probably 500 different ways... full names, partial names, abbreviations, abbreviations with dots, abbreviations with dots and extra spaces, different cases...

Well, you can't answer 50 states just because you had 100000 people. What if the source of that data was a CA insurance query form where they must enter, specifically, their CA address. (I'm assuming you'd still get other states, but it might not be 50).

Re: Data Is Never Raw

#14

Post by rhacker » Thu, Jan 10, 2019, 9:15 PM UTC

LanceH wrote: post #4

danso wrote: post #2

Related: I’ve been trying to get away from describing data as “dirty”, which among other things, implies that there is an obviously better “clean” state for the data. I think it better to call data “complicated”, and rather than cleaning it, we have ways to simplify it for processing.

Have 100,000 people enter there address with a text field for the state and get back to me with how many states there are.

Did the people reply with 50 really get the correct answer? Is this making the assumption that there is a "known" list of states, that addresses are entered correctly, that we're talking about a specific country?

The only way we really know the answer to this question is by placing a lot of bias into our answer.