Csv isn’t going anywhere for better or worse.
Time to retire the CSV?
521–530 of 594 posts
Re: Time to retire the CSV?
#522Earlier quoted context omitted.
Non standard delimiters. Escaping delimiters in fields - sometimes with a \, sometimes doubled (""), sometimes not at all. Double new lines. Poor handling from standard CSV libraries. Either unable to read or unable to create for some downstream process.
That sounds like the problem of badly formatted CSV, not a problem with CSV per se. If you stick to one delimiter, and that delimiter is a comma, and escape the delimiter in the data with double-quotes around the entry, and escape double quotes with two double-quotes, well, you have written CSV that is correct and looks correct and will be parsed correctly by literally every CSV parser.
Pretty much the rfc. https://datatracker.ietf.org/doc/html/rfc4180
Parsers are trickier if you want to be lenient, but exporters are dead simple.
Re: Time to retire the CSV?
#523"In favour of what?", that is the matter. CSV is a format more for humans and less for machines, but that is the use case: a format that is good enough to be compiled by humans and read by machines. At the moment there aren't many alternatives.
What are all these hypothetical humans doing with CSV files? I've worked with them a lot, and the only time I read one manually is to find out why it isn't parsing properly. I'm not writing them by hand. I'm not extracting data from them by hand. What are other people doing?
Re: Time to retire the CSV?
#524Earlier quoted context omitted.
> access through cat/grep/awk, and easily load into any programming language Until the CSV fields contain commas themselves. Even if fields are surrounded by "".
Some people like me never generate CSVs like that. If someone sends me a CSV with quotes, I reject it and say "clean up your data". It's okay to say, "we're not going to allow escape characters in our CSV grammar. We're going to have a higher standard"
Not everyone has the option of rejecting it, though. Sometimes you need that data and have no control over how it's being generated.
Re: Time to retire the CSV?
#525Re: Time to retire the CSV?
#526The worst I've seen is CSV in which some of the fields contain CSV.
Re: Time to retire the CSV?
#527Let’s replace them with sqlite dbs.
Re: Time to retire the CSV?
#528I don't agree with giving up csvs until the following conditions are met: 1) A truly open format is available and accessible. Csvs are textfiles. There is no system around that cannot open a textfile. If the format is binary or requires patents or whatever, then it's a non-starter. 2) Applications have a speed increase from using csvs. To wit, I loved csvs because often they finish preparing much faster than a "forma…
> The author is not forced to use csvs.
But some people are. There are entire industries built around the exchange of CSV files, and the producers and consumers don't necessarily talk to each other.
Re: Time to retire the CSV?
#529"In favour of what?", that is the matter. CSV is a format more for humans and less for machines, but that is the use case: a format that is good enough to be compiled by humans and read by machines. At the moment there aren't many alternatives.
What are all these hypothetical humans doing with CSV files? I've worked with them a lot, and the only time I read one manually is to find out why it isn't parsing properly. I'm not writing them by hand. I'm not extracting data from them by hand. What are other people doing?
Mainly Ctrl+F and diffs, in my case.
Sometimes you need to look up a single code in a 50 MB file. And sometimes you need a quick check to see if one line or a million lines changes.
It's "exception not the rule" type stuff... but it sure comes in handy to be able to check this stuff quick with basic text tools than have to run it through some binary parser. Same as JSON. But unlike protobufs for example.
Re: Time to retire the CSV?
#530Earlier quoted context omitted.
Can you give an example? The rules for CSV files are so simple I'm struggling to imagine a case where something looks correct but in fact isn't correct.
Me, a naive idiot: CSV is simple I will write my own exporter because I am clever Me, 20 minutes later: Heh that was easy I am a genius Me, 21 minutes later: Unicode is ruining my life T_T Don't get me wrong, I really like CSV because it's so primitive and works so well if you are disciplined about it. But it's easy to get something working on a small dataset and forget all the other possibilities only to faceplant a…
How does Unicode present a problem?
In UTF-8, the byte for a comma and a quote only exist as their characters. They don't exist as parts of multibyte sequences, by design.
If you have Unicode problems, then you have Unicode problems, but they wouldn't seem to be CSV problems...? Unless you're being incredibly sloppy in your programming and outputting double-byte UTF-16 strings surrounded by single-byte commas and quotes or something...?