Earlier quoted context omitted.
None of them seem all that conducive to source control or merging. Any good format for that?
Don't put data in source control; use a database.
And put that under source control.
Why isn’t there a decent file format for tabular data?
211–220 of 355 posts
Re: Why isn’t there a decent file format for tabular data?
#212> Columns are separated by \u001F (ASCII unit separator) > Rows are separated by \u001E (ASCII record separator) That's a nightmare to try to edit yourself in a text editor? I'd rather just have basically TSV, but with every value always quoted, always UTF-8. Quotes escaped with backslashes, backslashes escaped with backslashes, and that's it. Any binary allowed between the quotes. I deal with CSVs all day every day.…
Excellent posts. Coincidentally couple weeks ago while evaluating a HTTP response for a web service, I noticed that for tabular data, CSV is much more optimal than JSON; yet there is lack of HTTP header support for CSV responses that could provide clients with supplementary information in order to keep parsers adaptable.
If you have a copy of the said RFC, would like to refer.
Re: Why isn’t there a decent file format for tabular data?
#213Seems like the problem here is there is several high quality and well-developed formats, but the author and the commenters here dismiss them because of the different trade-offs they make. csv -- Simple for simple use cases, text-based, however many edge cases, feature lacking etc xlsx -- Works in excel, ubiquitous format with a standard, however complicated and missing scientific features sqlite -- Designed for relat…
I've seen ubiquitous use of tab-separated value files instead of csv, as as simpler format without quoting support and a restriction that your data fields can't contain tabs or newlines, which (unlike commas) is okay for many scenarios.
[deleted]
Re: Why isn’t there a decent file format for tabular data?
#214Parquet is a wonderful file format and is a dream to work with compared to CSV. Parquet embeds the schema in the footer metadata, so the query engines don't need to guess what the column names / data types are. Parquet used to be poorly supported, but now it's well supported by almost all languages. You can even view Parquet files in text editors now, but that's not something I've ever needed ( https://blog.jetbrains…
It's it possible to diff a parquet file?
I would load the parquet files in Python Pandas and do the diff you want using Pandas.
Re: Why isn’t there a decent file format for tabular data?
#215Seems like the problem here is there is several high quality and well-developed formats, but the author and the commenters here dismiss them because of the different trade-offs they make. csv -- Simple for simple use cases, text-based, however many edge cases, feature lacking etc xlsx -- Works in excel, ubiquitous format with a standard, however complicated and missing scientific features sqlite -- Designed for relat…
I've seen ubiquitous use of tab-separated value files instead of csv, as as simpler format without quoting support and a restriction that your data fields can't contain tabs or newlines, which (unlike commas) is okay for many scenarios.
But that’s sort of the problem with csv.
You never really know which rules your csv files has. Many .csv files are indeed tab separated.
Re: Why isn’t there a decent file format for tabular data?
#216[dead]
Re: Why isn’t there a decent file format for tabular data?
#217Earlier quoted context omitted.
So… not a text editor then, right?
So vim with a plugin isn't a text editor?
I just mean, if you _require_ plugins in order to be able to edit the content, then the content can't easily be described as text. It is fine to use a specialized application to edit a file of a non-text format, I have nothing against that, but you have then left the realm of text editor.
As an example of what I mean, if someone wrote a vim plugin that allowed a user to interact with a sqlite file and change the schema or edit raw row values from vim, it could be a really valuable and useful plugin. But the presence or absence of a plugin for some given text editor doesn't change whether a given format is generally considered a format suitable for being edited in a text editor. What it does instead is convert vim into an editor of non-text files.
I admit that the proposed file format is much closer to being editable in a text editor than a binary format such as sqlite, but the fact that the characters cannot be typed without special functionality suggests that the format is not suitable for a text editor.
Re: Why isn’t there a decent file format for tabular data?
#218Earlier quoted context omitted.
So… not a text editor then, right?
A Unicode text editor is not an ascii text editor either.
Actually, I think that it is, if it supports UTF-8 without BOM and does not try to do such things like convert quotation marks into non-ASCII quotation marks automatically, etc.
However, using a proper ASCII text editor would be better, if you do not want Unicode, to avoid many of the problems with Unicode if you are loading an unknown file or copying unknown stuff using clipboard, etc, which may be homoglyphs, reverse text direction override, etc.
(I use vim with exclusively ASCII-only mode, and do not use a Unicode locale, on my computer.)
Re: Why isn’t there a decent file format for tabular data?
#219> There doesn’t seem to be anything that is reasonably space efficient, simple and quick to parse and text based (not binary) so you can view and edit it with a standard editor.
> XML and Javascript are tree structures and not suitable for efficiently storing tabular data (plus other issues).
You can certainly be efficient with json(net). See:
Notice how they are separate objects:
{'name': 'foo', 'age': 2}
{'name': 'cat, 'age': 6}
Compress it if you need compact.
There's also UBF, but it never saw much traction: https://ubf.github.io/ubf/ubf-user-guide.en.html#specificati...
Re: Why isn’t there a decent file format for tabular data?
#220Earlier quoted context omitted.
You are limited to the basic types (int, floating point, string, and blob), however. I can somewhat get behind the opinionated argument for not needing more specific types like most common language types, but not the lack of a date type.
> but not the lack of a date type. i've also found this to be truly bizarre. even more bizarre than not actually respecting (via coercing or error) to the specified type...why even have types, then?
[deleted]