Perhaps we “just” need Excel and Numbers to support reading .parquet files, and come up with a CLI tool to view them like head/tail/awk? Yes, they still cannot be read as plain text so functionality is quite restricted with respect to CLI pipelines
Why isn’t there a decent file format for tabular data?
271–280 of 355 posts
Re: Why isn’t there a decent file format for tabular data?
#272I think it’s because csv is good enough. All the standards I’ve seen haven’t been worth the effort to implement. So since csv, with all its flaws, is good enough it crowds out other open standards. People complain about it, but it’s not really much of a challenge to use csv. I’d also prefer it over the crap (rdf, xml, even schemad json) proposed by people who value more structure. It’s easier for me to just make clea…
CSV has more problems than just quoting: one problem is that excel is so commonly used to open CSV files that various weirdnesses in excel have essentially become part of the CSV format. For example, if a value looks like a formula, then the only way to get excel to treat it as text is to put a single quote in front of it, "'=not a formula". This in turn means that its common for values in a CSV to begin with a singl…
> - Can be extended as needed with new type specifiers.
That's often not a good thing, at least if that allowance is given to third parties, as it wildly opens up the format to effective incompatibilities (fragmenting the ecosystem) and is a never ending source of security issues.
Re: Why isn’t there a decent file format for tabular data?
#273Earlier quoted context omitted.
I think CSV is crappy because commas are so common in real data. For almost all scenarios I've had to work with, I'd have been perfectly happy with TSV where literal Tab was a disallowed character. No escaping histrionics required.
Pipes are usually pretty safe too.
Depends entirely on context. I did consulting in wholesale and supplier data very often contained pipes as content separators (e.g. for product categories, sizes, etc.).
Re: Why isn’t there a decent file format for tabular data?
#274Earlier quoted context omitted.
But that’s sort of the problem with csv. You never really know which rules your csv files has. Many .csv files are indeed tab separated.
Gotta wonder why the format isn't just a column separator char, a row separator char, and then all the data guaranteed not to have those two chars. Then you could save the thing by finding any two chars that aren't used in the data. I guess this is why we have a zillion formats.
> Gotta wonder why the format isn't just a column separator char, a row separator char
That's what the article proposes.
Re: Why isn’t there a decent file format for tabular data?
#275Earlier quoted context omitted.
Gotta wonder why the format isn't just a column separator char, a row separator char, and then all the data guaranteed not to have those two chars. Then you could save the thing by finding any two chars that aren't used in the data. I guess this is why we have a zillion formats.
Then you can't edit or view it in a normal text editor, which is part of the appeal of CSV.
Maybe normal text editors should support the separators.
Re: Why isn’t there a decent file format for tabular data?
#276> Why can’t we have a format where Does Excel support it? No? Then that's the end of that. Excel is tabular data to all non developers. The formats supported by Excel are the whole thing. And if we're inventing a CSV-like format that uses a more convenient character than quotes and commas, maybe jumping to a non-displayable non-typeable character isn't the best? Honestly, if I were inventing a table format, I'd use a…
Works well until some joker comes along and puts or in your data :)
Re: Why isn’t there a decent file format for tabular data?
#277Earlier quoted context omitted.
But that’s sort of the problem with csv. You never really know which rules your csv files has. Many .csv files are indeed tab separated.
It’s a very meta-level problem. The main problem with CSV is “this is CSV” is an ambiguous statement. One could commit to a rule set, and give it a new name. Probably, someone has done this already.
This is why you say RFC 4810 CSV
Re: Why isn’t there a decent file format for tabular data?
#278SQLite is the answer.
Re: Why isn’t there a decent file format for tabular data?
#279> it could be called something like “unicode separated value”
There is. USV uses Unicode Record Separator and Unicode Field Separator.
And data handling tools such as Miller offer USV:
https://miller.readthedocs.io/en/latest/file-formats/#csvtsv...
Re: Why isn’t there a decent file format for tabular data?
#280> Columns are separated by \u001F (ASCII unit separator) > Rows are separated by \u001E (ASCII record separator) That's a nightmare to try to edit yourself in a text editor? I'd rather just have basically TSV, but with every value always quoted, always UTF-8. Quotes escaped with backslashes, backslashes escaped with backslashes, and that's it. Any binary allowed between the quotes. I deal with CSVs all day every day.…
Excellent posts. Coincidentally couple weeks ago while evaluating a HTTP response for a web service, I noticed that for tabular data, CSV is much more optimal than JSON; yet there is lack of HTTP header support for CSV responses that could provide clients with supplementary information in order to keep parsers adaptable. If you have a copy of the said RFC, would like to refer.
Don't know the parent poster's RFC... but there is a RFC 4810 which attempts to define CSV and when I looked was what Excel generated.