Live data from Hacker News

Why isn’t there a decent file format for tabular data?

successfulsoftware.net

21–30 of 355 posts

Re: Why isn’t there a decent file format for tabular data?

#21

I literally don't get why JSON is bad: [{row1}, {row2}, {row3}] The fact that it can do more is in no way a negative. Can even make a limited JSON parser with reduced capabilities. And with JSON can do more definitions like header names vs column names vs just arrays of arrays.

From TFA:

>> XML and Javascript are tree structures and not suitable for efficiently storing tabular data (plus other issues).

Re: Why isn’t there a decent file format for tabular data?

#22

I don't think anyone wants tabular data. They want to ingest it into a system so they can query it, or join it, or aggregate it. They want to get rid of tabular data as quickly as possible Data at the boundary must be validated, a file format can't do that for your. Semantics is harder than syntax and you can't push every problem to that level.

yeah, um, my life is in tables (stats, finance, etc).

while csv is less than optimal, it's always worked out in practice for me (because we still have to validate anything of significant size and nothing finance or city/nation sized should generally be edited by hand anyway, and errors are everywhere independent of format).

honestly, my bigger nightmare has been people trying to put things into something like xml...

edit: not that there aren't huge problems with the csv/excel/database software and interaction: generally I just end up having to assume everything is text and defining type information my end afterwards on successful parsing

Re: Why isn’t there a decent file format for tabular data?

#23

I literally don't get why JSON is bad: [{row1}, {row2}, {row3}] The fact that it can do more is in no way a negative. Can even make a limited JSON parser with reduced capabilities. And with JSON can do more definitions like header names vs column names vs just arrays of arrays.

Doesn't have a great int64 number story, no native dates / date-times. If you want named tuples, then the names need to go everywhere, otherwise it's a serialization mechanism on top of JSON.

Re: Why isn’t there a decent file format for tabular data?

#24

I don't think anyone wants tabular data. They want to ingest it into a system so they can query it, or join it, or aggregate it. They want to get rid of tabular data as quickly as possible Data at the boundary must be validated, a file format can't do that for your. Semantics is harder than syntax and you can't push every problem to that level.

We've had XML for decades and the default interop is still tabular data - probably in large part because unless you go into non-standard extensions, tabular data is the easiest thing to get from a database query.

Spreadsheets haven't exactly gone away either.

Re: Why isn’t there a decent file format for tabular data?

#25

I literally don't get why JSON is bad: [{row1}, {row2}, {row3}] The fact that it can do more is in no way a negative. Can even make a limited JSON parser with reduced capabilities. And with JSON can do more definitions like header names vs column names vs just arrays of arrays.

I think the problem is that there is no clear standard or convention for how to do this. Indeed there are many ways to represent tabular data. Most large software projects probably have at least one bespoke file format that they use internally. The trouble is the lack of such a format that everyone agrees on.

Re: Why isn’t there a decent file format for tabular data?

#26
> Why can’t we have a format where

Does Excel support it? No? Then that's the end of that.

Excel is tabular data to all non developers. The formats supported by Excel are the whole thing.

And if we're inventing a CSV-like format that uses a more convenient character than quotes and commas, maybe jumping to a non-displayable non-typeable character isn't the best?

Honestly, if I were inventing a table format, I'd use a strict subset of HTML5 tables. Strip it down as hard as possible to the bare minimal elements - no unneeded close tags. The only real flaw is that their encoding of whitespace and escaped chars are verbose and ugly.

    
    
    
        Example workbook
        
        
    
    
        
    Persons
    NameAddress
    Alice123 Fake St, Faketon FA 12345
    Bob789 Any Pl, Anytown AY 54321
    

    Cars
    MakeModel
    BuickLeSabre
    PontiacVibe
    

Re: Why isn’t there a decent file format for tabular data?

#27
post #10

> Columns are separated by \u001F (ASCII unit separator) > Rows are separated by \u001E (ASCII record separator) Or, how about columns separated by \u002C, and rows separated by \u000A. And, for bonus points, we can even define unambiguous ways of escaping those two characters so that they CAN appear within column values, if we wanted to, and not tell people that our encoding format is totally stupid and that they ne…

> OP's proposal is equally "highly sub-optimal for the job" for exactly the same imaginary reasons they dislike the currently available encoding formats, but they don't seem to realize it.

This is a really unfair appraisal in a bunch of different ways.

Removing the ability to embed record delimiters, for example, means you can process the records in parallel. That’s a massive improvement all by itself.

Stating that their reasons are “imaginary” is just a needless insult, apart from being wrong. Why be like that?

Re: Why isn’t there a decent file format for tabular data?

#28

I don't think anyone wants tabular data. They want to ingest it into a system so they can query it, or join it, or aggregate it. They want to get rid of tabular data as quickly as possible Data at the boundary must be validated, a file format can't do that for your. Semantics is harder than syntax and you can't push every problem to that level.

And the data in the system is… almost always tabular? Either Excel (which runs half the world) or RDB/SQL (which runs the other half).

Re: Why isn’t there a decent file format for tabular data?

#29
post #19

> No escaping. If you want to put \u001F or \u001E in your data – tough you > can’t. Use a different format. > It would be reasonably compact, efficient to parse and easy to manually edit > (Notepad++ shows the unit separator as a ‘US’ symbol). Is it me or it won't be human readable because of the lack of new lines?

I rarely want to see tabular data in a human-readable format. It is always the most tedious way to approach it. My go-to is Excel/LibreOffice Calc. This approach is at least tolerable to edit in a text editor, while something like the OpenDocument Spreadsheet format or the Excel format is impenetrable.

Re: Why isn’t there a decent file format for tabular data?

#30
post #26

> Why can’t we have a format where Does Excel support it? No? Then that's the end of that. Excel is tabular data to all non developers. The formats supported by Excel are the whole thing. And if we're inventing a CSV-like format that uses a more convenient character than quotes and commas, maybe jumping to a non-displayable non-typeable character isn't the best? Honestly, if I were inventing a table format, I'd use a…

it will also be at least 2x bigger on disk for no reason.
Post reply on HN