Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why isn’t there a decent file format for tabular data?

successfulsoftware.net

Post reply on HN
Print view View original

241–250 of 355 posts

Re: Why isn’t there a decent file format for tabular data?

#241

Post by m_eiman »

lordnacho wrote: post #233
wodenokoto wrote: post #215

Earlier quoted context omitted.

But that’s sort of the problem with csv. You never really know which rules your csv files has. Many .csv files are indeed tab separated.

Gotta wonder why the format isn't just a column separator char, a row separator char, and then all the data guaranteed not to have those two chars. Then you could save the thing by finding any two chars that aren't used in the data. I guess this is why we have a zillion formats.

Then you can't edit or view it in a normal text editor, which is part of the appeal of CSV.

Re: Why isn’t there a decent file format for tabular data?

#243

Post by hiyer »

BFLpL0QNek wrote: post #209
mountainriver wrote: post #174

Earlier quoted context omitted.

Arrow is really the future here

Isn’t Apache Arrow an in memory format that the various DataFrame libraries can standardise on to interact with each other? inter-process communication (IPC)? My understanding is your raw data on disk is still a format such as Parquet, but when you load that Parquet in to your application it’s stored as Arrow in-memory for processing?

Arrow also has its own on-disk format called Feather - https://arrow.apache.org/docs/python/feather.html

Re: Why isn’t there a decent file format for tabular data?

#244

Post by jarofgreen »

Fundamentally, why are people so concerned about about hand editing tabular data?

The options for that are pretty terrible. Only editing an existing value in one cell is ok. Everything else - adding new rows, new columns and making sure the file format and column structure is correct - is all terrible. This gets doubly so if you stick it in git and start getting messy pull requests.

I would carefully question what use case you have that this is so important. If you truly do have such a use case, I'd politely suggest tabular data maybe isn't what you need.

Once you remove that requirement, binary solutions for tabular data are possible again and several good options have already been pointed out in other comments.

Re: Why isn’t there a decent file format for tabular data?

#245

Post by mro_name »

fiddlerwoaroof wrote: post #232
SkyPuncher wrote: post #194

Earlier quoted context omitted.

Most csv utilities support an alternative delimiter. If I need to edit a file by hand, I'll typically pick an uncommon character for the delimiter (pipe "|" works well since it's uncommon). For me, that pretty much entirely eliminates any of the pain with CSV.

More tools should use the ASCII unit and field separator characters intended for this purpose: https://ronaldduncan.wordpress.com/2009/10/31/text-file-form...

wow, the solution hides in clear sight right before our eyes. Once again.

Re: Why isn’t there a decent file format for tabular data?

#246

Post by elmolino89 »

human_person wrote: post #37

Have you see tiledb? https://tiledb.com/data-types/dataframes My team is currently transitioning from HDF5 to tiledb for genomics data.

TileDB-VCF does work very well. Which types of data stored as HDF5s you are ingesting into TileDB?

Re: Why isn’t there a decent file format for tabular data?

#248

Post by mftb »

asperous wrote: post #126

Seems like the problem here is there is several high quality and well-developed formats, but the author and the commenters here dismiss them because of the different trade-offs they make. csv -- Simple for simple use cases, text-based, however many edge cases, feature lacking etc xlsx -- Works in excel, ubiquitous format with a standard, however complicated and missing scientific features sqlite -- Designed for relat…

As best I can tell no one has mentioned Recutils[0]?

It is a little bizarre that csv has never really been nailed down, but yea it's all about trade-offs.

[0]https://www.gnu.org/software/recutils/

Re: Why isn’t there a decent file format for tabular data?

#249

Post by eru »

hoseja wrote: post #247

Why are all these "separated"? Sounds like null-terminated C strings. Wouldn't it be better to put the dimension in a header and then just dump the contents unseparated? Why the need to view them in simple text editors?

What you suggested works for fixed width fields only, doesn't it?

Re: Why isn’t there a decent file format for tabular data?

#250

Post by atoav »

lordnacho wrote: post #233
wodenokoto wrote: post #215

Earlier quoted context omitted.

But that’s sort of the problem with csv. You never really know which rules your csv files has. Many .csv files are indeed tab separated.

Gotta wonder why the format isn't just a column separator char, a row separator char, and then all the data guaranteed not to have those two chars. Then you could save the thing by finding any two chars that aren't used in the data. I guess this is why we have a zillion formats.

I guess if you use an strange character nobody has on their keyboard, you could also just make it a binary format for efficiency
Post reply on HN