Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Friends don't let friends export to CSV

kaveland.no

Post reply on HN
Print view View original

161–170 of 459 posts

Re: Friends don't let friends export to CSV

#161

Post by gxs »

prepend wrote: post #85

This article seems written by someone who never had to work with diverse data pipelines. I work with large volumes of data from many different sources. I’m lucky to get them to send csv. Of course there are better formats, but all these sources aren’t able to agree on some successful format. Csv that’s zipped is producible and readable by everyone. And that makes is more efficient. I’ve been reading these “everyone i…

Agree.

Not saying csv doesn’t have its issues, but I don’t think the author made a convincing argument.

A lot of the issues the author brought up didn’t sound that bad and/or it sounds like he never looked at the source data first.

If you’re doing work with large datasets, I think it’s a good practice to at least go and look at the source data briefly to see what to expect.

This will give you a good idea of the format it outputs, data types, some domain context, etc. or some combination thereof and I don’t think it even takes that long.

Also, it reminds me of the arguments against excel in a way. Most people know what a csv is, more or less how to open it, and don’t need too much context when discussing the file. Someone will quickly understand if you tell them the file isn’t delimited properly right away. These are pros that shouldn’t be taken for granted.

Again, I’m not saying csv doesnt have issues or that there aren’t better alternatives, simply that I didn’t find this particular argument convincing.

Re: Friends don't let friends export to CSV

#162

Post by Dalewyn »

hn_throwaway_99 wrote: post #158
Dalewyn wrote: post #154

Earlier quoted context omitted.

>I do hope "Unicode Separated Values" takes off. It's essentially just a slight tweak to CSV where the delimiters are special unicode characters Commas can be typed by anyone on any keyboard and readable by anyone. Special Unicode Characters(tm) can't be typed by anyone on any keyboard and readable by noone. Convenience is a virtue.

I can't remember the last time I, or anyone I've ever worked with for that matter, ever typed up a CSV from scratch. The whole point of USV is that the delimiters can't normally be typed so you don't have to worry about escaping. USV supports displayable delimiters (see https://github.com/SixArm/usv ), so for the much more common case of editing an existing CSV in a text editor, you can just copy and paste.

I've valued the virtue of CSVs being readable by any text editor known to man, and I've occasionally edited them by hand. The pure simplicity of reading and typing commas trumps any value provided by more esoteric configurations.

As for escaping, that's for the subsequent programmers (which could also be me) to figure out. If it is me, I'll deal with it because it keeps things simple.

Re: Friends don't let friends export to CSV

#164

Post by chaps »

lostlogin wrote: post #148
ryncewynd wrote: post #116

Earlier quoted context omitted.

Are there unicode characters specifically for delimiters? If Excel had a standardised "Save as USV" option it would solve so many issues for me. I get so many broken CSVs from third-parties

> Are there unicode characters specifically for delimiters? We could use the HL7 pipe ‘|’ and all enjoy that hell.

My worst nightmare was a semicolon-delimited file. Where one of the columns had hand-typed street names - without quotes.. so "WELLS" was often "WE;;S".

Since it was the only column like that, the # of columns to the left of the annoying column and the # on the right would always stay the same. So it was pretty easy to clean.

Re: Friends don't let friends export to CSV

#165

Post by jgord »

whakim wrote: post #130
prepend wrote: post #85

This article seems written by someone who never had to work with diverse data pipelines. I work with large volumes of data from many different sources. I’m lucky to get them to send csv. Of course there are better formats, but all these sources aren’t able to agree on some successful format. Csv that’s zipped is producible and readable by everyone. And that makes is more efficient. I’ve been reading these “everyone i…

To me this criticism feels excessive. It feels like the author is describing their frustrations with internal usage of CSVs - there's no mention of customers and non-technical stakeholders at all. I think it goes without saying that Parquet files and other non-human-readable formats are a nonstarter when working with external stakeholders and the last paragraph makes that clear - if the end-user wants CSV, give them…

btw, xsv has solved most of my problems dealing with 'large' 40GB csv files

Re: Friends don't let friends export to CSV

#166

Post by prepend »

whakim wrote: post #130
prepend wrote: post #85

This article seems written by someone who never had to work with diverse data pipelines. I work with large volumes of data from many different sources. I’m lucky to get them to send csv. Of course there are better formats, but all these sources aren’t able to agree on some successful format. Csv that’s zipped is producible and readable by everyone. And that makes is more efficient. I’ve been reading these “everyone i…

To me this criticism feels excessive. It feels like the author is describing their frustrations with internal usage of CSVs - there's no mention of customers and non-technical stakeholders at all. I think it goes without saying that Parquet files and other non-human-readable formats are a nonstarter when working with external stakeholders and the last paragraph makes that clear - if the end-user wants CSV, give them…

I deal with gig size csvs all the time and don’t have any performance issues. These aren’t huge files, but decent sized. And most are just a few megs and only thousands to millions of records.

Csv is not very performant, but it doesn’t matter for these use cases.

I’ll also add that I’m not working with the csvs, they are just I/o. So any memory issues are handled by the load process. I certainly don’t use csvs for my internal processes. Just for when someone sends me data or I have to send it back to them.

That being said my workstation is pretty big and can handle 10s of gigs of csv before I care. But that’s usually just for dev or debugging and anything that sticks around will be working with data in some proper store (usually parquet distributed across nodes).

Re: Friends don't let friends export to CSV

#167

Post by prepend »

chaps wrote: post #141
whakim wrote: post #130

Earlier quoted context omitted.

To me this criticism feels excessive. It feels like the author is describing their frustrations with internal usage of CSVs - there's no mention of customers and non-technical stakeholders at all. I think it goes without saying that Parquet files and other non-human-readable formats are a nonstarter when working with external stakeholders and the last paragraph makes that clear - if the end-user wants CSV, give them…

Why is this getting downvoted? They're right that the criticism is pretty excessive: "Maybe his next article will be about how airlines should speak Esperanto because English is such a flawed language. That’s a clever and unique observation." Hm.

I got a little snarky but I think the analogy holds.

Esperanto is a superior language to English. And English has many flaws.

Theoretically it would be better to have all pilots and airports learn an efficient language.

But it would be stupid and immature to seriously write a blog post about that, especially without talking about all the flaws in that plan.

Re: Friends don't let friends export to CSV

#168

Post by doctor_eval »

lostlogin wrote: post #148
ryncewynd wrote: post #116

Earlier quoted context omitted.

Are there unicode characters specifically for delimiters? If Excel had a standardised "Save as USV" option it would solve so many issues for me. I get so many broken CSVs from third-parties

> Are there unicode characters specifically for delimiters? We could use the HL7 pipe ‘|’ and all enjoy that hell.

It’s been years since I last worked with HL7. Isn’t there also ^ and ~ to deal with?

Hell indeed.

Re: Friends don't let friends export to CSV

#169

Post by prepend »

msla wrote: post #118
prepend wrote: post #85

This article seems written by someone who never had to work with diverse data pipelines. I work with large volumes of data from many different sources. I’m lucky to get them to send csv. Of course there are better formats, but all these sources aren’t able to agree on some successful format. Csv that’s zipped is producible and readable by everyone. And that makes is more efficient. I’ve been reading these “everyone i…

> Csv that’s zipped is producible and readable by everyone. And that makes is more efficient. If only CSV were CSV, as opposed to some form that's 80-90% CSV by line count with enough oddities to really make the parser ugly and hard to read. See, the sweet spot isn't something completely unstructured, because then you feel justified in throwing up (your hands) and declaring defeat. The sweet spot is a file that's suf…

There’s definitely hair to deal with and it’s a little messy, but it’s never a blocker.

But I’d say the error rate is actually very low, maybe .1-1% and nowhere near 10-20% of data being messed up.

Re: Friends don't let friends export to CSV

#170

Post by fbdab103 »

ryncewynd wrote: post #116
hn_throwaway_99 wrote: post #111

Earlier quoted context omitted.

Totally agree. His arguments are basically "performance!" (which is honestly not important to 99% of CSV export users) and "It's underderspecified!" And while I can agree with the second, at least partly, in the real world the spec is essentially "Can you import it to Excel?". I'm amazed at how much programmers can discount "It already works pretty much everywhere" for the sake of more esoteric improvements. All that…

Are there unicode characters specifically for delimiters? If Excel had a standardised "Save as USV" option it would solve so many issues for me. I get so many broken CSVs from third-parties

My significantly bigger beef would be all of the auto-formatting Excel does to mangle data. Excel loves to turn entries into dates.

Human genes had to be renamed so as to avoid this Excel features.

Post reply on HN