Essential CSV shell tools:
csvtk: https://bioinf.shenwei.me/csvtk/
gawk: https://www.gnu.org/software/gawk/manual/html_node/Comma-Sep...
awk: https://github.com/onetrueawk/awk?tab=readme-ov-file#csv
A love letter to the CSV format
81–90 of 711 posts
Re: A love letter to the CSV format
#82The argument against JSON isn't very compelling. Adding a name to every field as they do in their strawman example isn't necessary. Compare this CSV field1,field2,fieldN "value (0,0)","value (0,1)","value (0,n)" "value (1,0)","value (1,1)","value (1,n)" "value (2,0)","value (2,1)","value (2,n)" To the directly-equivalent JSON [["field1","field2","fieldN"], ["value (0,0)","value (0,1)","value (0,n)"], ["value (1,0)","…
The flexibility of JSON is a downside when you just want to stream large volumes of row-oriented tabular data
Re: A love letter to the CSV format
#83The argument against JSON isn't very compelling. Adding a name to every field as they do in their strawman example isn't necessary. Compare this CSV field1,field2,fieldN "value (0,0)","value (0,1)","value (0,n)" "value (1,0)","value (1,1)","value (1,n)" "value (2,0)","value (2,1)","value (2,n)" To the directly-equivalent JSON [["field1","field2","fieldN"], ["value (0,0)","value (0,1)","value (0,n)"], ["value (1,0)","…
> Because CSV is so simple, it's common for them to avoid using a parsing/encoding library. A but unfair to compare CSV without parser library to JSON with library.
Essentially nobody uses JSON without a library, but tons of people (maybe even most people) use CSV without a library.
Part of the problem here is standards. There's a TON of encoding variations all using the same .csv extension. Making a library that can accurately detect exactly which one is correct is a big problem once you leave the handful of most common variants. If you are doing subfield encoding, you are almost certainly on your own with decoding at least part of your system.
JSON has just one standard and everyone adheres to that standard which makes fast libraries possible.
Re: A love letter to the CSV format
#84I greatly prefer TSV over CSV. https://en.wikipedia.org/wiki/Tab-separated_values
The problem with TSV is what are you going to do about quotes. Some fields might contain them [1] or they might be needed to store fields with tabs inside them.
Because of this in order to read a plain simple TSV (fields separated by tabs, nothing more) with the Python csv module [2] you need to set the quote character to an improbable value, say € (using the euro sign because HN won't let me use U+1F40D), or just parse it by hand, e.g. row.split('\t').
Re: A love letter to the CSV format
#85The argument against JSON isn't very compelling. Adding a name to every field as they do in their strawman example isn't necessary. Compare this CSV field1,field2,fieldN "value (0,0)","value (0,1)","value (0,n)" "value (1,0)","value (1,1)","value (1,n)" "value (2,0)","value (2,1)","value (2,n)" To the directly-equivalent JSON [["field1","field2","fieldN"], ["value (0,0)","value (0,1)","value (0,n)"], ["value (1,0)","…
The flexibility of JSON is a downside when you just want to stream large volumes of row-oriented tabular data
If you want to stream large volumes of row-oriented data, you aren't reading yourself and you should be using a binary format which is going to be significantly smaller (especially for numeric data).
Re: A love letter to the CSV format
#86Earlier quoted context omitted.
Your comma isn't my comma. French systems use the comma as a decimal point for numbers and we use semicolons to separate fields in CSV files.
No, french systems also use comma to separate fields in CSV files. Excel uses semicolon to separate fields in France, meaning it generates semicolon-separated files rather than comma-separated files. It's not the fault of CSV that Excel changes which file format it uses based on locale.
It's even worse than that. Office on my work computer is set to the English language, but my locale is French and so is my Windows language. It's saving semicolon-separated CSV files with the comma as a decimal point.
I need to uncheck File > Option Advanced > Use system separators and set the decimal separator to a dot to get Excel to generate English-style CSV files with semicolon-separated values. I can't be bothered to find out where Microsoft moved the CSV export dialog again in the latest version of Office to get it to spit out comma-separated fields.
Point is, CSV is a term for a bunch of loosely-related formats that depends among other things on the locale. In other words, it's a mess. Any sane file format either mandates a canonical textual representation for numbers independent of locale (like JSON) or uses binary (like BSON).
Re: A love letter to the CSV format
#87Anyone with a love of CSV hasn't been asked to deal with CSV-injection prevention in an enterprise setting, without breaking various customer data formats.
There's a dearth of good resources about this around the web, this is the best I've come across: https://georgemauer.net/2017/10/07/csv-injection.html
Re: A love letter to the CSV format
#88CSV is ever so elegant but it has one fatal flaw - quoting has "non-local" effects, i.e. an extra or missing quote at byte 1 can change the meaning of a comma at byte 1000000. This has (at least) two annoying consequences: 1. It's tricky to parallelise processing of CSV. 2. A small amount of data corruption can have a big impact on the readability of a file (one missing or extra quote can bugger the whole thing up).…
That would be solved by using the ASCII control chars Record Separator / Unit Separator! I don't get how this is not widely used as standard.
I remembered seeing a comment like this before, and...
comment: https://news.ycombinator.com/item?id=26305052
comment: https://news.ycombinator.com/item?id=39679662
"ASCII Delimited Text – Not CSV or Tab Delimited Text" post [2014]: https://news.ycombinator.com/item?id=7474600
same post [2024]: https://news.ycombinator.com/item?id=42100499
comment: https://news.ycombinator.com/item?id=15440801
(...and many more.) "This comes up every single time someone mentions CSV. Without fail." - top reply from burntsushi in that last link, and it remains as true today as in 2017 :D
You're not wrong though, we just need some major text editor to get the ball rolling and start making some attempts to understand these characters, and the rest will follow suit. We're kinda stuck at a local optimum which is clearly not ideal but also not troublesome enough to easily drum up wide support for ADSV (ASCII Delimiter Separated Values).
Re: A love letter to the CSV format
#89What isn't fun about CSV is quickly written parsers and serializers repeatedly making the common mistake of not handling, or badly handling, quoting.
For a long time I was very wary of CSV until I learnt Python and started using it's excellent csv standard library module.
Re: A love letter to the CSV format
#90I think I understand the point being made, but all this reliance on text-based data means we require proper agreement on text encodings, etc. I don't think it's very useful for number-based data anyway, it's a massively bloated way to store float32s for instance and usually developers truncate the data losing about half of the precision in the process.
For numerical data, nothing beats packing floats into blobs.