Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How to share your data effectively

caitlinrivers.com

Post reply on HN
Print view View original

81–90 of 127 posts

Re: How to share your data effectively

#81

Post by hmottestad »

acqq wrote: post #78
hmottestad wrote: post #70

Earlier quoted context omitted.

Nope. Not a joke. Lots of love from all over the world: http://lod-cloud.net/ The number one difference between RDF and CSV (TSV in you case) is that it's understandable by a computer. CSV is simple parsable, while RDF comes with all the meta data the computer needs.

Everything is readable by computer, if you have the program which reads it. For tabular data nobody needs the name of each field in every line. For non-tabular data, why not using JSON?

Understandable.

RDF is understandable by computers. It brings it's semantics along.

CSV is only syntax. It doesn't even support datatypes.

JSON is simply a data format serialisation. RDF can be serialised to JSON or to XML if you wish.

Re: How to share your data effectively

#82

Post by BHSPitMonkey »

crazygringo wrote: post #29

As long as the PDF contains text (and isn't just scans of paper), it's usually not TOO hard to select and copy the relevant tables/text, paste into a text editor, and use regex's to transform it into whatever you want (CSV, SQL insert statements, etc.) Since there's no universal data format (e.g. the data equivalent of PDF), I pretty much assume that whatever format data comes in, I'll have to be doing regex transfor…

Tables or columns are actually almost always a nightmare to copy and paste from a PDF. I have never had it come out sanely.

Re: How to share your data effectively

#83

Post by hmottestad »

Homunculiheaded wrote: post #80
hmottestad wrote: post #49

RDF, RDF, RDF, RDF, RDF, RDF, RDF. Yes. Please use RDF. And please use a common vocabulary. If you want to know how to do this? Have a look at the UK governments open data portal (which uses RDF): data.gov.uk And their SPARQL (query) endpoint: http://data.gov.uk/sparql

RDF took Prolog clauses, restricted them to triplets, and made their representation dramatically more verbose. Then it removed the reasoning engine part of logic programming and replaced it with OWL which is by default computationally intractable. I have tried for many, many years to learn to appreciate RDF and semantic technologies in general, but everytime I get involved it screams of a problem desperately looking…

JSON doesn't really support datatypes. So not really useful for sending data to someone else. Even simple things such as float vs. double will be a challenge. Not to mention dates or locations.

Pity you haven't gotten to like it. It's being tested on a lot of stuff, some sticks some doesn't.

And OWL has a number of subsets for whatever complexity you feel like enjoying.

Re: How to share your data effectively

#84

Post by BrianEatWorld »

droithomme wrote: post #55

It seems like most federal and state agencies nowadays make much data - data that they are forced by law to make public - only available in pdf files. The reason for this seems to be to prevent as much as possible use and analysis of the data, while technically complying with public disclosure requirements. It is often extremely difficult to get public data in a usable form out of most public web sites, and the ways…

Having worked for the US government, trying to collect trade and production data from other governments and private entities, I always assumed this was the case when I received PDFs. Interestingly, interest groups were always eager to give you the data in whatever format you requested...

Re: How to share your data effectively

#85

Post by BrianEatWorld »

cookiecaper wrote: post #37
JustARandomGuy wrote: post #6

Use a flexible file format. My preference is .csv, because it can be read by almost any program. I'll tolerate .xls, but I'm not pleased with .xlsx (not everyone uses Excel!). And please, please, please do not use pdf. Why is .xls tolerated but not .xlsx? XLS and XLSX are both Excel formats, but XLS is a binary blob while XLSX is just a zip containing xml files. I'd much rather parse xml files (XLSX) than the binary…

My experience is that open-source implementations like LibreOffice have much better support for the older file formats. docx/xlsx/etc may be a "zip of XML files" but it doesn't really matter if the original data is stored as XML or a blob when it comes to the end user's rendering experience -- it's about how the system can interpret that data. And the interpretations of the new file formats are mostly severely lackin…

To be fair, isn't Libre is also pretty terrible at handling more basic formats like csvs?

I still have yet to find a good workaround for the "numbers as text" issue aside from switching to tsvs.

Re: How to share your data effectively

#87

Post by maxerickson »

crazygringo wrote: post #29

As long as the PDF contains text (and isn't just scans of paper), it's usually not TOO hard to select and copy the relevant tables/text, paste into a text editor, and use regex's to transform it into whatever you want (CSV, SQL insert statements, etc.) Since there's no universal data format (e.g. the data equivalent of PDF), I pretty much assume that whatever format data comes in, I'll have to be doing regex transfor…

For pdfs that are coming from some sort of consistent system, the conversion tools in xpdf work pretty well.

Re: How to share your data effectively

#88

Post by gamblor956 »

droithomme wrote: post #55

It seems like most federal and state agencies nowadays make much data - data that they are forced by law to make public - only available in pdf files. The reason for this seems to be to prevent as much as possible use and analysis of the data, while technically complying with public disclosure requirements. It is often extremely difficult to get public data in a usable form out of most public web sites, and the ways…

The law requires state and federal agencies to make the data public. It does not require them to make the data useful for a specific use case.

But generally, the reasons PDF are used is: (1) PDF signals (internally) that the file has been examined from sensitive, confidential, or non-public content, and no longer needs further modification; (2) PDF is a long-lived standard with plenty of free viewers; and (3) PDF guarantees print fidelity across all known viewers, which is something most office software (including online suites) can't even promise between minor version updates.

Re: How to share your data effectively

#90

Post by pvaldes »

Several good advices, but missing the real problem here. Of course you CAN extract text from a pdf and you can extract a table from excel. A few lines of perl or your favourite language and you have it.

Data are worthless if you can't trust it. Plain CSV are easy to read but easy to change, even on the fly. Pdf can be changed also, but is not so easy and if someone makes a subtile change in a number the error don't propagates by all the pages reaching the totals like in excel. Excel macros/formulas can be a source of headaches.

So to share your data use a format that: 1-you can trust (reasonably). 2-all your other reasons go here...

And if this format is full of nested tables, and you find difficult to extract the info from those tables, don't throw out the format, ask for help instead.

PDF can support passwords, is very compact, not so easy to change on the fly and can be encrypted. Maybe not the best, but not the worst of the available tools, in my opinion.

Post reply on HN