Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why isn’t there a decent file format for tabular data?

successfulsoftware.net

Post reply on HN
Print view View original

201–210 of 355 posts

Re: Why isn’t there a decent file format for tabular data?

#201

Post by kragen »

The proposal is TSV but using ^_ instead of ^I (tab) to separate columns and ^^ instead of ^M (CR) or ^J (LF) to separate rows. So instead of being unable to put ^I and ^J in your table cells you are now unable to put ^_ and ^^ in your table cells.

That sounds exactly as good, or bad, as TSV. So, okay? Sure, do that if you like, that sounds fine.

Emacs TAGS files, Info files, and BABYL mailboxes https://quimby.gnus.org/notes/BABYL similarly use weird control characters as delimiters.

Re: Why isn’t there a decent file format for tabular data?

#202

Post by thayne »

MobiusHorizons wrote: post #140
thayne wrote: post #118

Earlier quoted context omitted.

> That's a nightmare to try to edit yourself in a text editor? You just need a text editor that can support thia format.

So… not a text editor then, right?

So vim with a plugin isn't a text editor?

Re: Why isn’t there a decent file format for tabular data?

#203

Post by PeterisP »

asperous wrote: post #126

Seems like the problem here is there is several high quality and well-developed formats, but the author and the commenters here dismiss them because of the different trade-offs they make. csv -- Simple for simple use cases, text-based, however many edge cases, feature lacking etc xlsx -- Works in excel, ubiquitous format with a standard, however complicated and missing scientific features sqlite -- Designed for relat…

I've seen ubiquitous use of tab-separated value files instead of csv, as as simpler format without quoting support and a restriction that your data fields can't contain tabs or newlines, which (unlike commas) is okay for many scenarios.

Re: Why isn’t there a decent file format for tabular data?

#204

Post by jonas21 »

Banana699 wrote: post #190
asperous wrote: post #126

Seems like the problem here is there is several high quality and well-developed formats, but the author and the commenters here dismiss them because of the different trade-offs they make. csv -- Simple for simple use cases, text-based, however many edge cases, feature lacking etc xlsx -- Works in excel, ubiquitous format with a standard, however complicated and missing scientific features sqlite -- Designed for relat…

The author doesn't like any of those tradeoffs and wishes to make another one, what's the problem with that ? You don't believe the design space is exhaustively explored by the designs and protocols you mentioned, do you? there is always another local optimum to be found.

The problem is that the author states that none of these existing formats are "decent" and falls back on shallow dismissals like "Don’t even get me started on Excel’s proprietary, ghastly binary format."

Re: Why isn’t there a decent file format for tabular data?

#205

Post by riffraff »

SkyPuncher wrote: post #194
asperous wrote: post #126

Seems like the problem here is there is several high quality and well-developed formats, but the author and the commenters here dismiss them because of the different trade-offs they make. csv -- Simple for simple use cases, text-based, however many edge cases, feature lacking etc xlsx -- Works in excel, ubiquitous format with a standard, however complicated and missing scientific features sqlite -- Designed for relat…

Most csv utilities support an alternative delimiter. If I need to edit a file by hand, I'll typically pick an uncommon character for the delimiter (pipe "|" works well since it's uncommon). For me, that pretty much entirely eliminates any of the pain with CSV.

tab-separated-value has never betrayed me! I think it's the default postgresql file export too.

Re: Why isn’t there a decent file format for tabular data?

#206

Post by PeterisP »

zoomablemind wrote: post #180
leeoniya wrote: post #153

Earlier quoted context omitted.

> but not the lack of a date type. i've also found this to be truly bizarre. even more bizarre than not actually respecting (via coercing or error) to the specified type...why even have types, then?

What's so special about having a named type for datetime? User will still need to call functions to manipulate the dates. If only for the default display and import?

For dates, having a specific date type instead of a text field is required to have proper behavior when sorting and efficient storage/data transfer.

It's also important to have date-related functions on the DB server side, so that you can use them in filtering data before it gets sent over to the user code running on the client, to avoid unnecessary data transfer and allow proper use of indexes in optimizing it.

Also, it is nice if a DB engine can perform the equivalent of `WHERE year(date)=2021` without actually running that function on every date, but rather automatically optimize it to an index lookup of `WHERE date between '2021-01-01' and '2021-12-31'`.

Re: Why isn’t there a decent file format for tabular data?

#207

Post by 6yyyyyy »

garaetjjte wrote: post #142
AdamH12113 wrote: post #34

I never understood why the ASCII separator characters aren't used more. It seems like we're one simple text editor feature away from having easy display and modification. Is there some historical reason for not doing that?

Because they don't have standardized escaping, and when you are rolling your own ad-hoc escaping scheme you can just use printable separator anyway.

Really? Then I wonder what the ASCII character coded 0x10 is for?

Re: Why isn’t there a decent file format for tabular data?

#208

Post by sanderjd »

lumost wrote: post #196
asperous wrote: post #126

Seems like the problem here is there is several high quality and well-developed formats, but the author and the commenters here dismiss them because of the different trade-offs they make. csv -- Simple for simple use cases, text-based, however many edge cases, feature lacking etc xlsx -- Works in excel, ubiquitous format with a standard, however complicated and missing scientific features sqlite -- Designed for relat…

I've often wondered what would happen if there was a standard text editor plugin for dealing with parquet and co. It seems like these formats are disliked, as they are difficult to inspect - but there really isn't any reason UTF-8 bytes arranged in a large sequence (aka CSV) should be any easier to read except for editor support. Sure writes would be slower, but I'd expect most users wouldn't care on modern hardware.

Yeah I don't really understand the downside to a format like parquet here. "Less ubiquitous" seems to be the only one in the parent comment's list?

Re: Why isn’t there a decent file format for tabular data?

#209

Post by BFLpL0QNek »

mountainriver wrote: post #174
BFLpL0QNek wrote: post #144

Earlier quoted context omitted.

You want to search for “DataFrame” libraries. Another commenter mentioned Spark, Panda’s is another popular one, not used it but think it’s lighter weight where Spark is more for large distributed computation even though can run locally. There’s a bunch of these tools which lets you treat parquet files as tables doing joins, aggregations etc.

Arrow is really the future here

Isn’t Apache Arrow an in memory format that the various DataFrame libraries can standardise on to interact with each other? inter-process communication (IPC)?

My understanding is your raw data on disk is still a format such as Parquet, but when you load that Parquet in to your application it’s stored as Arrow in-memory for processing?

Re: Why isn’t there a decent file format for tabular data?

#210

Post by zzo38computer »

6yyyyyy wrote: post #207
garaetjjte wrote: post #142

Earlier quoted context omitted.

Because they don't have standardized escaping, and when you are rolling your own ad-hoc escaping scheme you can just use printable separator anyway.

Really? Then I wonder what the ASCII character coded 0x10 is for?

That is what I considered too, actually.
Post reply on HN