Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Friends don't let friends export to CSV

kaveland.no

Post reply on HN
Print view View original

341–350 of 459 posts

Re: Friends don't let friends export to CSV

#341

Post by scbrg »

iainmerrick wrote: post #297
hn_throwaway_99 wrote: post #183

Earlier quoted context omitted.

> The first time we came across CSV format we likely typed it in notepad by hand. Again, I'm not saying CSVs aren't edited by hand in a text editor, I'm saying they aren't created from scratch in a text editor, even by beginners. USVs are easy to edit in a text editor, too, and I tried viewing and editing USVs with a couple different fonts and had no problems.

If the separators can't easily be typed, how do you add a new cell?

Nobody can type up a GIF image, or Word document in a Notepad, yet files of both those formats exist. The answer obviously is tooling. If a format with sane separators was common, so would editors that could edit that format be.

Re: Friends don't let friends export to CSV

#342

Post by smcin »

prepend wrote: post #166
whakim wrote: post #130

Earlier quoted context omitted.

To me this criticism feels excessive. It feels like the author is describing their frustrations with internal usage of CSVs - there's no mention of customers and non-technical stakeholders at all. I think it goes without saying that Parquet files and other non-human-readable formats are a nonstarter when working with external stakeholders and the last paragraph makes that clear - if the end-user wants CSV, give them…

I deal with gig size csvs all the time and don’t have any performance issues. These aren’t huge files, but decent sized. And most are just a few megs and only thousands to millions of records. Csv is not very performant, but it doesn’t matter for these use cases. I’ll also add that I’m not working with the csvs, they are just I/o. So any memory issues are handled by the load process. I certainly don’t use csvs for my…

> That being said my workstation is pretty big and can handle 10s of gigs of csv before I care.

How much RAM do you have? What's the ratio of [smallest CSV file which bottlenecks]/[your RAM]?

Re: Friends don't let friends export to CSV

#345

Post by ratherbefuddled »

Ubiquity has a quality all of its own.

Yes CSV is a pain in many regards, but many of the difficulties with it arise from the fact that anybody can produce it with very little tool support - which is also the reason it is so widely used.

Recommending a decidedly niche format as an alternative is not going anywhere.

Re: Friends don't let friends export to CSV

#347

Post by nlitened »

echoangle wrote: post #245
GuB-42 wrote: post #97

As a French, there is another problem with CSV. In the French locale, the decimal point is the comma, so "121.5" is written "121,5". It means, of course, that the comma can't be used as a separator, so the semicolon is used instead. It means that depending whether or not the tool that exports the CSV is localized or not, you get commas or you get semicolons. If you are lucky, the tool that imports it speaks the same…

Does any tool seriously localize CSVs?

So many tools do that, even (most importantly) Excel does

Re: Friends don't let friends export to CSV

#348

Post by iainmerrick »

scbrg wrote: post #341
iainmerrick wrote: post #297

Earlier quoted context omitted.

If the separators can't easily be typed, how do you add a new cell?

Nobody can type up a GIF image, or Word document in a Notepad, yet files of both those formats exist. The answer obviously is tooling. If a format with sane separators was common, so would editors that could edit that format be.

I was responding to the GP's:

USVs are easy to edit in a text editor

I don't see how that's the case.

If a format with sane separators was common, so would editors that could edit that format be

Sure, but that's a hypothetical future editor, not something that currently exists.

Edit to add: I also disagree with "sane" in that context. New separators won't solve anything. You'll always need escaping/encoding/encapsulation; get that right and everything else follows. JSON is comma-separated and does just fine.

Re: Friends don't let friends export to CSV

#349

Post by Oioioioiio »

prepend wrote: post #85

This article seems written by someone who never had to work with diverse data pipelines. I work with large volumes of data from many different sources. I’m lucky to get them to send csv. Of course there are better formats, but all these sources aren’t able to agree on some successful format. Csv that’s zipped is producible and readable by everyone. And that makes is more efficient. I’ve been reading these “everyone i…

I worked plenty enough with 'diverse data pipelines' and most of them were shit due to other companies just not knowing how to work properly.

CSV created tons of issues regarding encoding, value separation etc.

I started talking to our customers and were able to define interfaces with better and aligned format. json made my life easier.

Re: Friends don't let friends export to CSV

#350

Post by fifilura »

rkaveland wrote: post #261

Author here. I see now that the title is too controversial, I should have toned that down. As I mention in the conclusion, if you're giving parquet files to your user and all they want to know is how to turn it into Excel/CSV, you should just give them Excel/CSV. It is, after all, what end users often want. I'm going to edit the intro to make the same point there. If you're exporting files for machine consumption, pl…

Congratulations for getting the article upvoted and don't be too hard on yorself.
Post reply on HN