> CSVs are built for humans, not machines
And that's why they're not going away.
Trying to add typing to them is probably missing the point that >95% of humans that use them don't need or want to understand that.
Time to retire the CSV?
341–350 of 594 posts
Re: Time to retire the CSV?
#342Earlier quoted context omitted.
You're comparing CSVs to other spreadsheet document formats. But a CSV is not a spreadsheet. A CSV is raw data. (It's data that is restricted to a shape that enables it to be easily imported into a spreadsheet—but data nevertheless.) As such, it should be compared to other data formats—e.g. YAML, JSON Lines, etc. These other data formats all win on your #2 against CSV, as CSV is actually horrible at parse-time vs. ot…
If you interpret "CSV" as purely comma seperated values then maybe. But in my bubble "CSV" means textfiles that are separated by some separator. Be it tabs, spaces, commas, or any other ASCII character. Some are more usable then others, if you have commas in your data then use tabs. If you have tabs use Form Feed or Record Separator or vertical tabs ... and so on. Of course this is not always applicable, since you so…
It seems quite common in some European countries to use semi-colons as the delimiter instead of commas (because they use commas as the decimal separator?), adding a new level of fun to parsing. In Easy Data Transform we count the number of commas, semi-colons, tabs in the file to make an educated guess as the delimiter (which the user can override).
Re: Time to retire the CSV?
#343Earlier quoted context omitted.
...or, you know, you could use the ASCII characters specifically defined for separating records and units. ;-)
If only I could type them on my keyboard. (I think this is a big part of why CSV is the way it is — people want to be able to hand-edit it, or at least hand-produce small test datasets to test the systems on the other end.)
The funny thing is you can type any character on a keyboard. It's the same weird disconnect about "text file formats are human-readable". Either way you need a tool that can render & read the format in question. It does't much matter what format you actually store the data in, because you don't read & write the bites directly.
Re: Time to retire the CSV?
#344Earlier quoted context omitted.
No, SQLite's dynamic data types would silently coerce data just like opening a CSV directly with Excel does. The advantage of CSV is that it's as accurate as your plain text representation of your data can be. Since binary data can be represented by character data, that's 100% accurate. As soon as you introduce a storage format that has made assumptions about the type of data being stored, you've lost flexibility. SQ…
Excel's default insistence on trying to coerce the data into what it expects the data to be is the source of many of the problems of people using CSV. Excel will even ignore quotes around values if it thinks that the value inside might be a number. I often work with CSV files that contain 17 digit IDs that use numeric characters "01233998742021925" which Excel will convert lossily into scientific notation rather than…
Excel trying to be helpful and silently converting data is a source of endless pain. In fact it is so bad, that geneticists have renamed some genes:
https://www.theverge.com/2020/8/6/21355674/human-genes-renam...
Re: Time to retire the CSV?
#345If you want to design yet another ‘better’ file format, go to it. But CSV files are fine the way they are, thanks. If I need to preserve metadata, there are many tried and true options available to me.
Re: Time to retire the CSV?
#346Earlier quoted context omitted.
The issue is CSV is not compatible with CSV. It's not possible to write a spec that covers all CSV files in the world. CSV means things that are mutually incompatible in the less common cases, and the only way to really parse them correctly is to know which variant generated it. But you can't even tag that variant in the file by your criteria, as existing CSV parsers won't understand it.
Couldn't tools that read CSV files scan them first and see which variant best matches the file? The questions arise about which end of line character(s) are used and how double-quotes and commas are handled. There can't be that many ways to escape them, and there are three sets of end of line characters mentioned - just see which one is used (i.e. don't assume only \n if you run into \r\n or \r alone). The assumption…
>Couldn't tools that read CSV files scan them first and see which variant best matches the file?
Yes. And my software does that. But it is always going to be a guess which the user needs to be able to override.
Re: Time to retire the CSV?
#347CSV is bad - but convenience is hard to beat. It's the same thing with excel. People abuse it, but you just can't beat the fact that your programmer, CEO, analyst, and secretary can all contribute to the same file.
At the end of the day the purpose of software is to serve users, not satisfy developers’ sense of perfection.
Re: Time to retire the CSV?
#348I don't agree with giving up csvs until the following conditions are met: 1) A truly open format is available and accessible. Csvs are textfiles. There is no system around that cannot open a textfile. If the format is binary or requires patents or whatever, then it's a non-starter. 2) Applications have a speed increase from using csvs. To wit, I loved csvs because often they finish preparing much faster than a "forma…
You're comparing CSVs to other spreadsheet document formats. But a CSV is not a spreadsheet. A CSV is raw data. (It's data that is restricted to a shape that enables it to be easily imported into a spreadsheet—but data nevertheless.) As such, it should be compared to other data formats—e.g. YAML, JSON Lines, etc. These other data formats all win on your #2 against CSV, as CSV is actually horrible at parse-time vs. ot…
> means that there's no way to parallelize CSV processing, because there's no way to read-ahead and "chunk" a CSV purely lexically.
There's a subset of CSV that forbids escapes that is super fast to parse. All fast CSV parsers I'm aware of take advantage of this subset. I try to never ever publish a CSV that has quotes, and always aim for a more restrictive grammar that is cleaner, better thought out data.
Re: Time to retire the CSV?
#349Earlier quoted context omitted.
"Self-describing formats like JSON Lines are big... but when you compress them, they go back to being small." For CSV file replacements, I'd expect something like "one JSON array per line, all values must be JSON scalars". In that case, it's not much larger than a CSV, especially one using quotes already for string values. But this demonstrates the problem with JSON for CSV, I suppose. Is each line an object? Is it w…
> But this demonstrates the problem with JSON for CSV, I suppose. Is each line an object? How is that not a problem with every data serialization format? It does me no real good if I have an XML schema and a corresponding file. If I don't know what those elements and attributes represent I'm not really any better off. It's not like JSON or XML can meaningfully be marshaled back into objects for use generically withou…
> That's why Microsoft got away with proprietary date formats in System.Text.Json.
What's proprietary in it? It follows ISO 8601-1:2019 and RFC 3339 according to the docs.
Re: Time to retire the CSV?
#350I know that there's an xkcd comic about this but I think that the best solution would be to create a new text based format with a new extension ie. "bsv" - better separated values. This new format would either be much stricter than csv or have a compulsory header that defines the file's format (like the text encoding, separator character etc).
If it is stricter, it would have one type of field separator that is not commas since some locales use them as decimal places (I'm looking at you, France) but something like '|'. It would insist that dates were iso8601. It could define how fields can be escaped and quoted - although I would prefer if quoting was kept to a minimum. The format should also allow for comments i.e # so that people can comment their datasets inside the same file.
Alternatively or in addition, it could have some header lines:
1) A header that defines the encoding, separator, decimal separator, quote character, escape character, line ending character, date format ...
2) A header that defines each column's name
3) A header that defines each column's data type and formatting
4) A header that defines each column's unit like m/s or kg - ok, this is a bit of a stretch but it would be great to have.
or some variation of the above.
Fundamentally, this bsv format would still be csv and most programs would still be able to read it with the parsers that already exist or be quickly adapted to read it. It could still be easily edited by hand but the metadata would be present.
I suspect that this is just a pipe dream because people would find hundreds of ways to break it but toml took off and that didn't exist so long ago.