Earlier quoted context omitted.
None of them seem all that conducive to source control or merging. Any good format for that?
Don't put data in source control; use a database.
Laughs in Lisp
Code is Data would like to have a word
Why isn’t there a decent file format for tabular data?
151–160 of 355 posts
Re: Why isn’t there a decent file format for tabular data?
#152> Columns are separated by \u001F (ASCII unit separator) > Rows are separated by \u001E (ASCII record separator) That's a nightmare to try to edit yourself in a text editor? I'd rather just have basically TSV, but with every value always quoted, always UTF-8. Quotes escaped with backslashes, backslashes escaped with backslashes, and that's it. Any binary allowed between the quotes. I deal with CSVs all day every day.…
If every value is always UTF-8, then you can't embed arbitrary binary, since arbitrary bytes aren't necessarily valid UTF-8.
Sure you can, here is example:
0101010010000111101010101
Re: Why isn’t there a decent file format for tabular data?
#153Earlier quoted context omitted.
The latest version of SQLite has a STRICT command to enforce the data types. This option is set per table, but even in a STRICT table you can specify the type of some columns as ANY if you want to allow any type of data in that column (this is not the same meaning of ANY in non-strict tables).
You are limited to the basic types (int, floating point, string, and blob), however. I can somewhat get behind the opinionated argument for not needing more specific types like most common language types, but not the lack of a date type.
> but not the lack of a date type.
i've also found this to be truly bizarre. even more bizarre than not actually respecting (via coercing or error) to the specified type...why even have types, then?
Re: Why isn’t there a decent file format for tabular data?
#154> Columns are separated by \u001F (ASCII unit separator) > Rows are separated by \u001E (ASCII record separator) That's a nightmare to try to edit yourself in a text editor? I'd rather just have basically TSV, but with every value always quoted, always UTF-8. Quotes escaped with backslashes, backslashes escaped with backslashes, and that's it. Any binary allowed between the quotes. I deal with CSVs all day every day.…
If every value is always UTF-8, then you can't embed arbitrary binary, since arbitrary bytes aren't necessarily valid UTF-8.
We’re talking about a tabular data file format. If you want to include arbitrary binary, use a binary data file. Or base64 encoded data. Most datasets you’d use data like this for are small enough to fit into memory, so let’s not get carried away.
(I happen to use tab delimited files to store data that can’t fit into memory, but that’s okay too)
Re: Why isn’t there a decent file format for tabular data?
#155Seems like the problem here is there is several high quality and well-developed formats, but the author and the commenters here dismiss them because of the different trade-offs they make. csv -- Simple for simple use cases, text-based, however many edge cases, feature lacking etc xlsx -- Works in excel, ubiquitous format with a standard, however complicated and missing scientific features sqlite -- Designed for relat…
Ther are tons of billion dollar companies that have entire systems utilizing csv and xlsx tubular data for mission critical processes lol
Re: Why isn’t there a decent file format for tabular data?
#156Earlier quoted context omitted.
I think CSV is crappy because commas are so common in real data. For almost all scenarios I've had to work with, I'd have been perfectly happy with TSV where literal Tab was a disallowed character. No escaping histrionics required.
I prefer commas because I can see them over tabs. I spend zero time escaping commas because the libraries and read and write with (usually pandas but pretty much everything) escape for me. So unless I’m manually building csvs myself it’s a non-issue and certainly not histrionics.
I prefer tabs because my data can often have commas in it. Seeing tabs isn’t an issue as I also have invisible characters visible in all of my editors.
But having your delimiter not be allowed in the record (as in \t is disallowed), makes parsing so much easier. CSV is a bear to parse because you have to read each value from a buffer to handle the quoting.
line.strip().split(‘\t’)
Re: Why isn’t there a decent file format for tabular data?
#157The article's proposal seems worse than tab separated.
Not having escaping is unacceptable of course, and once you add escaping you might as well use the standard tab and newline characters as delimiters (and the \t, \n, \\ escapes plus \N for SQL NULL), resulting in a file that is properly formatted by the terminal and text editors and just works with Unix tools and most spreadsheet/database importers unlike a file with the weird separators the article proposes.
Re: Why isn’t there a decent file format for tabular data?
#158I think it’s because csv is good enough. All the standards I’ve seen haven’t been worth the effort to implement. So since csv, with all its flaws, is good enough it crowds out other open standards. People complain about it, but it’s not really much of a challenge to use csv. I’d also prefer it over the crap (rdf, xml, even schemad json) proposed by people who value more structure. It’s easier for me to just make clea…
CSV is fine. If you care about edge cases, implement RFC 4180: https://www.rfc-archive.org/getrfc.php?rfc=4180 If you don't, then split each line on ",". Problem solved. If you find tab delimited easier to read (as I do), then check out the IANA spec for TSV files: https://www.iana.org/assignments/media-types/text/tab-separa... It's easier to parse than CSV. Unfortunately, you have to decide how to handle newlines an…
> If you don't, then split each line on ",". Problem solved.
What? Real data contains '"' quote character too. That doesn't work.
Re: Why isn’t there a decent file format for tabular data?
#159Don't know why csv cant be done in parallel by splitting across lines.
Re: Why isn’t there a decent file format for tabular data?
#160HDF5 is often used in scientific computing for this. https://en.wikipedia.org/wiki/Hierarchical_Data_Format
HDF5 has some limitations that make it suboptimal for cloud based storage systems. Zarr overcomes these limitations for array data and Parquet overcomes these limitations for tabular data.
The OP wants a text based format, he doesn't care about what is optimal.