Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How to share your data effectively

caitlinrivers.com

Post reply on HN
Print view View original

71–80 of 127 posts

Re: How to share your data effectively

#71

Post by brudgers »

droithomme wrote: post #55

It seems like most federal and state agencies nowadays make much data - data that they are forced by law to make public - only available in pdf files. The reason for this seems to be to prevent as much as possible use and analysis of the data, while technically complying with public disclosure requirements. It is often extremely difficult to get public data in a usable form out of most public web sites, and the ways…

Any format other than PDF and the large numbers of citizens will not have the proper tool to access it - e.g. a person using a public computer in a library.

It's not malice even if it hampers outside researchers. Any format other than printed (e.g. PDF) is going to require the user to deal with the meta-data in order to extract useful information - that's just the nature of data.

Re: How to share your data effectively

#72

Post by omni »

dav-id wrote: post #13
mrtriangle wrote: post #5

Earlier quoted context omitted.

I agree with you, I actually had a client who demanded that the data in their project be done in excel (with multiple files ranging from 1-3 gb in size!). Both myself and my partner tried exhaustively to use a database or even JSON/XML solutions but the client kept repeating "in the real world business is done in excel!" Needless to say it was a nightmare.

I don't have the full picture here but by moving away from an Excel format you are also potentially taking away a lot of control they have themselves over managing the data and working the data which Excel is extremely good at with little to no programming.

Your point is true generally, but I have yet to use a version of Excel that can handle multiple gigabytes of data without serious performance degradation and probably a lot of crashes.

Re: How to share your data effectively

#73

Post by prodigal_erik »

shrikant wrote: post #28
onemorepassword wrote: post #20

My preference is .csv, because it can be read by almost any program. Except the one most used for it: Excel. Excel sucks at csv, especially if the recipients have various internationalized versions. It can't deal properly with multi-line strings and spaces, but especially the internationalization is hell (semi-colons instead of commas, decimal separators, date formats). Many times have I dealt with CSV-exports that c…

I've always found the C in CSVs to be needlessly limiting. I'm not aware of any tool that contextualise commas in addresses and other "free text" fields that might form part of the data dump/extract. For this reason, I am strongly in favour of pipe-separated values in such files. The probability that a data extract has a pipe symbol in any data field is quite small (in my experience, it's been 0 so far). The first th…

Then you run into ASCII art some end user made, or pre-Unicode text from Scandinavia where 0x7C was the code point of a letter. Commas make it pretty obvious which tools are unusably broken, where very rare characters let these bugs go undetected far too long.

Re: How to share your data effectively

#74

Post by mirkules »

qw wrote: post #52
eli wrote: post #44

Earlier quoted context omitted.

You left out the absolute worst bit: Excel tries to "helpfully" guess the correct data format for each cell in a CSV file. This can lead to silent data loss. Long strings of numbers are converted to a Numeric type that only stores a certain number of significant figures. The string "1-5" is converted to a Date field with the value "5-Jan"

You can specify the field types when opening the file.

Indeed. I don't know what version of Excel the parent is using, but Excel for Mac 2010 handles csv files just fine, even ones that have weird delimiters. I'd love to see an example of a non-readable csv file.

Edit: I haven't tried different Unicodes yet, so granted that may not work.

Re: How to share your data effectively

#75

Post by amboar »

brudgers wrote: post #71
droithomme wrote: post #55

It seems like most federal and state agencies nowadays make much data - data that they are forced by law to make public - only available in pdf files. The reason for this seems to be to prevent as much as possible use and analysis of the data, while technically complying with public disclosure requirements. It is often extremely difficult to get public data in a usable form out of most public web sites, and the ways…

Any format other than PDF and the large numbers of citizens will not have the proper tool to access it - e.g. a person using a public computer in a library. It's not malice even if it hampers outside researchers. Any format other than printed (e.g. PDF) is going to require the user to deal with the meta-data in order to extract useful information - that's just the nature of data.

They could publish the data in multiple formats, PDF and something actually useful if you're worried about someone at a library not having the tools. The point is, don't cripple the data from an analysis perspective.

Re: How to share your data effectively

#76

Post by bliker »

...and horrors of math equations and columns in PDF. We should really come up with some light format where style is separated from data. Make a universal converter from doc and xls files and save world from data stuck in PDFs.

Re: How to share your data effectively

#77

Post by SilasX »

droithomme wrote: post #55

It seems like most federal and state agencies nowadays make much data - data that they are forced by law to make public - only available in pdf files. The reason for this seems to be to prevent as much as possible use and analysis of the data, while technically complying with public disclosure requirements. It is often extremely difficult to get public data in a usable form out of most public web sites, and the ways…

Wait, how come they release newly-public data to traders on financial markets in easily computer-parseable format like JSON, but PDFs with tables for so much of this stuff?

You'd almost think it's an issue of priorities.

Re: How to share your data effectively

#78

Post by acqq »

hmottestad wrote: post #70
acqq wrote: post #62

Earlier quoted context omitted.

Is that a joke? RDF and SPARQL look exactly like something even mother has problem to love. http://en.wikipedia.org/wiki/SPARQL My personal preference is, whenever there's a lot of records to be processed, a tab separated UTF-8 text file. (I admit I am biased as I like to use Perl for first passes).

Nope. Not a joke. Lots of love from all over the world: http://lod-cloud.net/ The number one difference between RDF and CSV (TSV in you case) is that it's understandable by a computer. CSV is simple parsable, while RDF comes with all the meta data the computer needs.

Everything is readable by computer, if you have the program which reads it. For tabular data nobody needs the name of each field in every line.

For non-tabular data, why not using JSON?

Re: How to share your data effectively

#79

Post by jessriedel »

hmottestad wrote: post #69
jessriedel wrote: post #51

Earlier quoted context omitted.

Is this what you're talking about? http://en.wikipedia.org/wiki/Resource_Description_Framework I'm interested in knowing a little more about why you like it so much.

RDF is a graph based data format (serialised to XML). It is based on triples. Such as: Peter hasParent Paal. To query RDF, we use the graph query language SPARQL. You create a graph with variables in it, and the query engine will find subgraphs that match the graph. On top of this we can do reasoning (based on 1st order logics and datalog). With RDFS (simple) and OWL (advanced). And to bring all this together there a…

Hey this is really cool. Thanks.

Re: How to share your data effectively

#80

Post by Homunculiheaded »

hmottestad wrote: post #49

RDF, RDF, RDF, RDF, RDF, RDF, RDF. Yes. Please use RDF. And please use a common vocabulary. If you want to know how to do this? Have a look at the UK governments open data portal (which uses RDF): data.gov.uk And their SPARQL (query) endpoint: http://data.gov.uk/sparql

RDF took Prolog clauses, restricted them to triplets, and made their representation dramatically more verbose. Then it removed the reasoning engine part of logic programming and replaced it with OWL which is by default computationally intractable.

I have tried for many, many years to learn to appreciate RDF and semantic technologies in general, but everytime I get involved it screams of a problem desperately looking for a solution, and whose general case of the problem (reasoning about relations) was already solved better decades ago (and with the recent revived interested in logic/relational programming seems likely to improve soon).

If you want to represent a graph structure in your data why not sets of json objects? Sure you end up with an ad hoc vocabulary, but I've never seen a successful project make use of the RDF ideal of massively shared vocabularies.

Post reply on HN