Friends don't let friends export to CSV

Post by [deleted] » Tue, Mar 26, 2024, 9:26 PM UTC

hn_throwaway_99 wrote: post #111

prepend wrote: post #85

This article seems written by someone who never had to work with diverse data pipelines. I work with large volumes of data from many different sources. I’m lucky to get them to send csv. Of course there are better formats, but all these sources aren’t able to agree on some successful format. Csv that’s zipped is producible and readable by everyone. And that makes is more efficient. I’ve been reading these “everyone i…

Totally agree. His arguments are basically "performance!" (which is honestly not important to 99% of CSV export users) and "It's underderspecified!" And while I can agree with the second, at least partly, in the real world the spec is essentially "Can you import it to Excel?". I'm amazed at how much programmers can discount "It already works pretty much everywhere" for the sake of more esoteric improvements. All that…

Post by rossdavidh » Tue, Mar 26, 2024, 9:30 PM UTC

Too wrote: post #420

ultrarunner wrote: post #403

Earlier quoted context omitted.

Real world example of this that we just experienced: I work with a provider who offers CSV exports as the only way to access data. Recently, we found they were including unsanitized user input directly in fields. They weren't even quoting these fields. The platform "notified their quality assurance team ASAP" (like every other issue, we never heard back), but we had a deadline. This, of course, was a mess, but being…

This sounds like a problem that wouldn’t have existed in the first place if following a binary protocol with a standard format and using a proper serialization library. The issue comes from CSV files looking easy to generate by hand, when it in fact is not.

Post by jdougan » Tue, Mar 26, 2024, 9:33 PM UTC

wombatpm wrote: post #249

djbusby wrote: post #219

Earlier quoted context omitted.

ASCII has had field and record separators since like, forever. Wish we had kept using those.

No you don’t. It’s a holdover from when files were on tapes. The logic is all inverted too. Record separators were at the beginning of a record. Group and unit separators could then be nested. You really needed a spec for whatever document you were trying to makt

Post by fennecbutt » Tue, Mar 26, 2024, 9:39 PM UTC

prepend wrote: post #85

This article seems written by someone who never had to work with diverse data pipelines. I work with large volumes of data from many different sources. I’m lucky to get them to send csv. Of course there are better formats, but all these sources aren’t able to agree on some successful format. Csv that’s zipped is producible and readable by everyone. And that makes is more efficient. I’ve been reading these “everyone i…

Post by asddubs » Tue, Mar 26, 2024, 10:17 PM UTC

emj wrote: post #268

asddubs wrote: post #254

Earlier quoted context omitted.

I think a takeaway could also be not to give people options when making a human-readable format. "you always need quotes, they're not optional" solves the comma problem. "the delimiter is always a comma" solves the delimiter problem. json has also fared better than csv, I'd say.

That makes the delimiter "," which is ugly so someone is just going to use , instead and you are back to square one.