Data Is Never Raw
thenewatlantis.com
Data Is Never Raw
1–10 of 14 posts
Re: Data Is Never Raw
#2Related: I’ve been trying to get away from describing data as “dirty”, which among other things, implies that there is an obviously better “clean” state for the data. I think it better to call data “complicated”, and rather than cleaning it, we have ways to simplify it for processing.
Re: Data Is Never Raw
#3Previously: https://news.ycombinator.com/item?id=18735492
Re: Data Is Never Raw
#4Related: I’ve been trying to get away from describing data as “dirty”, which among other things, implies that there is an obviously better “clean” state for the data. I think it better to call data “complicated”, and rather than cleaning it, we have ways to simplify it for processing.
Have 100,000 people enter there address with a text field for the state and get back to me with how many states there are.
Re: Data Is Never Raw
#5Related: I’ve been trying to get away from describing data as “dirty”, which among other things, implies that there is an obviously better “clean” state for the data. I think it better to call data “complicated”, and rather than cleaning it, we have ways to simplify it for processing.
Have 100,000 people enter there address with a text field for the state and get back to me with how many states there are.
50 states, probably. ;)
Edit: assuming a very US-centric dataset...
Re: Data Is Never Raw
#6Earlier quoted context omitted.
Have 100,000 people enter there address with a text field for the state and get back to me with how many states there are.
50 states, probably. ;) Edit: assuming a very US-centric dataset...
50 states, represented probably 500 different ways... full names, partial names, abbreviations, abbreviations with dots, abbreviations with dots and extra spaces, different cases...
Re: Data Is Never Raw
#7Related: I’ve been trying to get away from describing data as “dirty”, which among other things, implies that there is an obviously better “clean” state for the data. I think it better to call data “complicated”, and rather than cleaning it, we have ways to simplify it for processing.
Semantic normalization?
Re: Data Is Never Raw
#8Earlier quoted context omitted.
50 states, probably. ;) Edit: assuming a very US-centric dataset...
50 states, represented probably 500 different ways... full names, partial names, abbreviations, abbreviations with dots, abbreviations with dots and extra spaces, different cases...
That data happens to take the form of human input. In that case the different ways in which participants choose to enter their address is part of what you are collecting. It does not suggest that the data is raw or dirty, you just gathered superfluous information as a side effect of your methodology. I think it is better described as overly complicated.
If the problem were the opposite it would also make sense to say that the data is too simple rather than too clean. That gets to what the article is saying; that all data collection is inherently biased.
Re: Data Is Never Raw
#9Related: I’ve been trying to get away from describing data as “dirty”, which among other things, implies that there is an obviously better “clean” state for the data. I think it better to call data “complicated”, and rather than cleaning it, we have ways to simplify it for processing.
Have 100,000 people enter there address with a text field for the state and get back to me with how many states there are.
There are definitely cases in which the possible values are ennumerable and known. I’m thinking more about situations in which a many-to-one relation, or other ambiguity is attempted to be collected. Such as coming across multiple values in a race/gender field in a police interaction database because multiple officers were involved. My problem with using “dirty” is that it is so ambiguous as to encompass relatively simple issues that can be translated via a lookup table, as well as messiness that actually reflects the messiness of the reality beyond the input form’s common case. In the latter case, there isn’t a “clean” situation as much as there is a rethinking of what you thought you were measuring.
Re: Data Is Never Raw
#10Related: I’ve been trying to get away from describing data as “dirty”, which among other things, implies that there is an obviously better “clean” state for the data. I think it better to call data “complicated”, and rather than cleaning it, we have ways to simplify it for processing.
There’s validated data types and unvalidated data types.
“Our logs are stored as newline-delimited JSON objects.” No, it’s improperly escaped so it’s actually a JSON-like string of unknown character encoding.
“This is a postal code field.” Nope, it’s a user-generated string that occasionally converges on something approximating a postal code.
“This is the user’s IP address.” Well I clearly see localhost in here so unless folks are browsing the site from our servers, this is “sometimes” the user’s IP address.