How to share your data effectively
caitlinrivers.com
How to share your data effectively
1–10 of 127 posts
Re: How to share your data effectively
#2It does seem to be a big challenge to make data available both in analyzed, "crunched" format for readers of a report (in which case PDF is usually fine), and in "raw" format for analysts and developers (in which case one of CSV flatfile, JSON or XML is usually the best format). Too often, people don't seem to recognize the difference.
The internet isn't just about software and protocols, it's also about the data. Sharing data is kind of the whole point of networking, so making your data available in a machine-parseable format is so important.
Re: How to share your data effectively
#3There is lots of bad advice in this article. OK maybe the article is shorter than I thought. Here, this is bad:
Use short variable names with no whitespace. Underscores are usually a safe bet, so instead of "Number of new tuberculosis cases" use "incident_tb"
While it might be clear to the domain experts, I had no idea that "incident_tb" meant "Number of new tuberculosis cases" other than that it was placed next to that text.
I think the message is right, but honestly if they are sending you data to parse in PDF, you better just give up now. That person will most likely not have seen a README file before in their life, they're probably not even trying.
Re: How to share your data effectively
#4I've used scraperwiki.com before to convert PDFs to "usable" data in cases like these. Yeah, it's a pain, but then it's available for everyone.
Re: How to share your data effectively
#5There is lots of bad advice in this article. OK maybe the article is shorter than I thought. Here, this is bad: Use short variable names with no whitespace. Underscores are usually a safe bet, so instead of "Number of new tuberculosis cases" use "incident_tb" While it might be clear to the domain experts, I had no idea that "incident_tb" meant "Number of new tuberculosis cases" other than that it was placed next to t…
I agree with you, I actually had a client who demanded that the data in their project be done in excel (with multiple files ranging from 1-3 gb in size!). Both myself and my partner tried exhaustively to use a database or even JSON/XML solutions but the client kept repeating "in the real world business is done in excel!" Needless to say it was a nightmare.
Re: How to share your data effectively
#6Use a flexible file format. My preference is .csv, because it can be read by almost any program. I'll tolerate .xls, but I'm not pleased with .xlsx (not everyone uses Excel!). And please, please, please do not use pdf.
Why is .xls tolerated but not .xlsx?
XLS and XLSX are both Excel formats, but XLS is a binary blob while XLSX is just a zip containing xml files. I'd much rather parse xml files (XLSX) than the binary blob of XLS.
Re: How to share your data effectively
#7There is lots of bad advice in this article. OK maybe the article is shorter than I thought. Here, this is bad: Use short variable names with no whitespace. Underscores are usually a safe bet, so instead of "Number of new tuberculosis cases" use "incident_tb" While it might be clear to the domain experts, I had no idea that "incident_tb" meant "Number of new tuberculosis cases" other than that it was placed next to t…
This can be helpful advice for many people. Why bash him? It's good advice. I would understand "incident_tb" in health context, and I'm 17. So a readme should be plenty, right?
Re: How to share your data effectively
#8Use a flexible file format. My preference is .csv, because it can be read by almost any program. I'll tolerate .xls, but I'm not pleased with .xlsx (not everyone uses Excel!). And please, please, please do not use pdf. Why is .xls tolerated but not .xlsx? XLS and XLSX are both Excel formats, but XLS is a binary blob while XLSX is just a zip containing xml files. I'd much rather parse xml files (XLSX) than the binary…
Likely because it's not a well known fact about XLSX and DOCX.
Re: How to share your data effectively
#9There is lots of bad advice in this article. OK maybe the article is shorter than I thought. Here, this is bad: Use short variable names with no whitespace. Underscores are usually a safe bet, so instead of "Number of new tuberculosis cases" use "incident_tb" While it might be clear to the domain experts, I had no idea that "incident_tb" meant "Number of new tuberculosis cases" other than that it was placed next to t…
I agree with you, I actually had a client who demanded that the data in their project be done in excel (with multiple files ranging from 1-3 gb in size!). Both myself and my partner tried exhaustively to use a database or even JSON/XML solutions but the client kept repeating "in the real world business is done in excel!" Needless to say it was a nightmare.
"Well, that explains why the business world is so broken"
Re: How to share your data effectively
#10Use a flexible file format. My preference is .csv, because it can be read by almost any program. I'll tolerate .xls, but I'm not pleased with .xlsx (not everyone uses Excel!). And please, please, please do not use pdf. Why is .xls tolerated but not .xlsx? XLS and XLSX are both Excel formats, but XLS is a binary blob while XLSX is just a zip containing xml files. I'd much rather parse xml files (XLSX) than the binary…
XLSX and the other OpenXML formats are generally speaking very easy to query, particularly as Microsoft provide their OpenXML SDK to do it with as well as other 3rd party tools provide similar functionality.