I hate CSV (but not as much as XML).
Most reasonably large CSV files will have issues parsing on another system.
A love letter to the CSV format
31–40 of 711 posts
Re: A love letter to the CSV format
#32CSV is ever so elegant but it has one fatal flaw - quoting has "non-local" effects, i.e. an extra or missing quote at byte 1 can change the meaning of a comma at byte 1000000. This has (at least) two annoying consequences: 1. It's tricky to parallelise processing of CSV. 2. A small amount of data corruption can have a big impact on the readability of a file (one missing or extra quote can bugger the whole thing up).…
That would be solved by using the ASCII control chars Record Separator / Unit Separator! I don't get how this is not widely used as standard.
Perhaps the popularity,or lack thereof? More often than not, the bad standard wins the long term market
Re: A love letter to the CSV format
#33Earlier quoted context omitted.
That would be solved by using the ASCII control chars Record Separator / Unit Separator! I don't get how this is not widely used as standard.
The _entire_ point of a CSV file is that it's fully human readable and write-able. The characters you mention could be used in a custom delimiter variant of the format, but at that point it's back to a binary machine format.
What do you mean? I just push the Record Separator key on my keyboard.
/s in case :)
Re: A love letter to the CSV format
#34CSV is ever so elegant but it has one fatal flaw - quoting has "non-local" effects, i.e. an extra or missing quote at byte 1 can change the meaning of a comma at byte 1000000. This has (at least) two annoying consequences: 1. It's tricky to parallelise processing of CSV. 2. A small amount of data corruption can have a big impact on the readability of a file (one missing or extra quote can bugger the whole thing up).…
Eh, all you're really saying is "I'm not using CSV. Instead I'm using my CSV." Except that's all that anybody does.
CSV can just as easily support escaping as any other format, but there is no agreement for a CSV format.
After all, a missed escape can just as easily destroy a JSON or XML structure. And parallel processing of text is already a little sketchy simply because UTF-8 exists.
Re: A love letter to the CSV format
#35CSV is ever so elegant but it has one fatal flaw - quoting has "non-local" effects, i.e. an extra or missing quote at byte 1 can change the meaning of a comma at byte 1000000. This has (at least) two annoying consequences: 1. It's tricky to parallelise processing of CSV. 2. A small amount of data corruption can have a big impact on the readability of a file (one missing or extra quote can bugger the whole thing up).…
That would be solved by using the ASCII control chars Record Separator / Unit Separator! I don't get how this is not widely used as standard.
Can't type them on a keyboard I guess, or generally work with them in the usual text-oriented tools? Part of the appeal of CSV is you can just open it up in Notepad or something if you need to. Maybe that's more a critique of text tools than it is of ASCII record separator characters.
Re: A love letter to the CSV format
#36Earlier quoted context omitted.
I don't get it - why the world, Excel can't just open the CSV, assume from the extension it's COMMA separated value and do the rest. It does work slightly better when importing, just a little.
Your comma isn't my comma. French systems use the comma as a decimal point for numbers and we use semicolons to separate fields in CSV files.
No, french systems also use comma to separate fields in CSV files. Excel uses semicolon to separate fields in France, meaning it generates semicolon-separated files rather than comma-separated files.
It's not the fault of CSV that Excel changes which file format it uses based on locale.
Re: A love letter to the CSV format
#37Excel hates CSV only if you don't use the "From text / csv" function (under the data tab).
For whatever reason, it flawlessly manages to import most CSV data using that functionality. It is the only way I can reliably import data to excel with datestamps / formats.
Just drag/dropping a CSV file onto a spreadsheet, or "open with excel" sucks.
Re: A love letter to the CSV format
#38I greatly prefer TSV over CSV. https://en.wikipedia.org/wiki/Tab-separated_values
TSV looks incredibly ugly when opened in a text editor unless all values are 7 characters or less.
Re: A love letter to the CSV format
#39CSV still quietly powers the majority of the world’s "data plumbing."
At any medium+ sized company, you’ll find huge amounts of CSVs being passed around, either stitched into ETL pipelines or sent manually between teams/departments.
It’s just so damn adaptable and easy to understand.
Re: A love letter to the CSV format
#40One thing that has changed the game with how I work with CSVs is ClickHouse. It is trivially easy to run a local database, import CSV files into a table, and run blazing-fast queries on it. If you leave the data there, ClickHouse will gradually optimize the compression. It's pretty magical stuff if you work in data science.