Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How to share your data effectively

caitlinrivers.com

Post reply on HN
Print view View original

121–127 of 127 posts

Re: How to share your data effectively

#121

Post by crntaylor »

TsiCClawOfLight wrote: post #7
yebyen wrote: post #3

There is lots of bad advice in this article. OK maybe the article is shorter than I thought. Here, this is bad: Use short variable names with no whitespace. Underscores are usually a safe bet, so instead of "Number of new tuberculosis cases" use "incident_tb" While it might be clear to the domain experts, I had no idea that "incident_tb" meant "Number of new tuberculosis cases" other than that it was placed next to t…

This can be helpful advice for many people. Why bash him? It's good advice. I would understand "incident_tb" in health context, and I'm 17. So a readme should be plenty, right?

> Why bash him?

*her

Re: How to share your data effectively

#122

Post by maxerickson »

BrianEatWorld wrote: post #85
cookiecaper wrote: post #37

Earlier quoted context omitted.

My experience is that open-source implementations like LibreOffice have much better support for the older file formats. docx/xlsx/etc may be a "zip of XML files" but it doesn't really matter if the original data is stored as XML or a blob when it comes to the end user's rendering experience -- it's about how the system can interpret that data. And the interpretations of the new file formats are mostly severely lackin…

To be fair, isn't Libre is also pretty terrible at handling more basic formats like csvs? I still have yet to find a good workaround for the "numbers as text" issue aside from switching to tsvs.

What do you mean by good workaround?

Specifying that the column is text during the import at least gets the data into the sheet.

Re: How to share your data effectively

#123

Post by abraxasz »

Yes, a million times yes. I'd like to emphasize the following points:

(2), (3), (4) are all tied to the same idea: Your data will be parsed by a script, not by me. I'm not going to use excel to analyse your data. I'm gonna import them into R/Python/... and then extract the features I'm interested in. I've been a graduate student in statistics for almost 2 years now, and when the data are correctly packaged, I've never had to use excel.

(5) Come on, I know that nobody likes writing README's. Heck, I don't like it myself. But do you really expect me to guess what your two-letters variable name means? And please, make your readme searchable, and short. The cdc data comes with a 200 pages manual explaining what the data is, and finding what you're looking for is a nightmare..

Re: How to share your data effectively

#124

Post by tom_b »

JustARandomGuy wrote: post #6

Use a flexible file format. My preference is .csv, because it can be read by almost any program. I'll tolerate .xls, but I'm not pleased with .xlsx (not everyone uses Excel!). And please, please, please do not use pdf. Why is .xls tolerated but not .xlsx? XLS and XLSX are both Excel formats, but XLS is a binary blob while XLSX is just a zip containing xml files. I'd much rather parse xml files (XLSX) than the binary…

I have had reasonable success with using the Apache POI project for the XLSX and XLS. You might want to consider looking at the community edition of Pentaho Data Integration at the 5.0 level and up. It includes POI for XLSX. I use it all the time to deal with random spreadsheets full of data I get from clinical data (hospital reports).

But I hate excel of any type for data. People randomly change formats and give me updated spreadsheets with 100K's of rows. Then seem surprised when their data doesn't link up with other data sources anymore . . .

Re: How to share your data effectively

#125

Post by BrianEatWorld »

maxerickson wrote: post #122
BrianEatWorld wrote: post #85

Earlier quoted context omitted.

To be fair, isn't Libre is also pretty terrible at handling more basic formats like csvs? I still have yet to find a good workaround for the "numbers as text" issue aside from switching to tsvs.

What do you mean by good workaround? Specifying that the column is text during the import at least gets the data into the sheet.

Yes, it gets data into the sheet, but means that you need extra steps to run calculations. There is no option to specify the column as a number during the import phase. So what takes one step in Excel takes a minimum of two in Libre.

Re: How to share your data effectively

#126

Post by maxerickson »

BrianEatWorld wrote: post #125
maxerickson wrote: post #122

Earlier quoted context omitted.

What do you mean by good workaround? Specifying that the column is text during the import at least gets the data into the sheet.

Yes, it gets data into the sheet, but means that you need extra steps to run calculations. There is no option to specify the column as a number during the import phase. So what takes one step in Excel takes a minimum of two in Libre.

I thought you were worried about leading zeros or something.

In calc 4.0 on XP, I can choose from a few options for each column and the default (called "Standard") results in a column that I can use in calculations (if I highlight the cells it shows the sum, in formulas, etc.).

If I specify the column is "Text", leading zeros are retained.

Post reply on HN