Friends don't let friends export to CSV • Hacker News Board

Re: Friends don't let friends export to CSV

#141

Post by chaps » Tue, Mar 26, 2024, 1:50 AM UTC

whakim wrote: post #130

prepend wrote: post #85

This article seems written by someone who never had to work with diverse data pipelines. I work with large volumes of data from many different sources. I’m lucky to get them to send csv. Of course there are better formats, but all these sources aren’t able to agree on some successful format. Csv that’s zipped is producible and readable by everyone. And that makes is more efficient. I’ve been reading these “everyone i…

To me this criticism feels excessive. It feels like the author is describing their frustrations with internal usage of CSVs - there's no mention of customers and non-technical stakeholders at all. I think it goes without saying that Parquet files and other non-human-readable formats are a nonstarter when working with external stakeholders and the last paragraph makes that clear - if the end-user wants CSV, give them…

Why is this getting downvoted? They're right that the criticism is pretty excessive:

  "Maybe his next article will be about how airlines should speak Esperanto because English is such a flawed language. That’s a clever and unique observation."

Hm.

Re: Friends don't let friends export to CSV

#142

Post by fshr » Tue, Mar 26, 2024, 1:54 AM UTC

sfRattan wrote: post #107

Gabriel54 wrote: post #101

Earlier quoted context omitted.

If I'm not mistaken this is pretty universal outside of the US (and maybe the UK).

Going by the Wikipedia article and included map, use of comma versus period as decimal separators is roughly an even split: https://en.wikipedia.org/wiki/Decimal_separator https://commons.wikimedia.org/wiki/File:DecimalSeparator.svg

There's definitely a big distribution disparity. 11 of the 15 most populous countries use the period for decimals.

Re: Friends don't let friends export to CSV

#143

Post by nradov » Tue, Mar 26, 2024, 1:54 AM UTC

thayne wrote: post #139

nradov wrote: post #66

Earlier quoted context omitted.

If both parties implement RFC 4180 and use a consistent character set encoding then I don't think there are actually any undefined behaviors. But in practice a lot of implementations are simply broken, including those from major tech companies that ought to know better.

And the application doesn't try to convert the cells into non-string data types like numbers, dates, etc.

Converting strings into other data types is out of scope for CSV, not really undefined behavior. The type conversions happen at a later stage of the import process.

Re: Friends don't let friends export to CSV

#144

Post by Larrikin » Tue, Mar 26, 2024, 1:57 AM UTC

sfRattan wrote: post #107

Gabriel54 wrote: post #101

Earlier quoted context omitted.

If I'm not mistaken this is pretty universal outside of the US (and maybe the UK).

Going by the Wikipedia article and included map, use of comma versus period as decimal separators is roughly an even split: https://en.wikipedia.org/wiki/Decimal_separator https://commons.wikimedia.org/wiki/File:DecimalSeparator.svg

Seems geographically split, but I wonder what is the actual population split is. Most of the top 10 population countries use the decimal separator. Only Brazil, Russia and Indonesia don't.

Maybe someone with a CSV of the world populations and a CSV of the countries broken down by their separator can do that comparison.

Re: Friends don't let friends export to CSV

#145

Post by randomsolutions » Tue, Mar 26, 2024, 1:59 AM UTC

I like the ping pong of one day an article being posted where everyone asks, "when/why did everything become so complicated", and then the next day something like this is posted.

Re: Friends don't let friends export to CSV

#146

Post by kerkeslager » Tue, Mar 26, 2024, 2:01 AM UTC

breadwinner wrote: post #98

The reason CSV is popular is because it is (1) super simple, and (2) the simplicity leads to ubiquity. It is extremely easy to add CSV export and import capability to a data tool, and that has come to mean that there are no data tools that don't support CSV format. Parquet is the opposite of simple. Even when good libraries are available (which it usually isn't), it is painful to read a Parquet file. Try reading a Pa…

> Google Protobuf is a lot closer than Parquet or Avro. But Protobuf is not a splitable format, which means it is not Big Data friendly, unlike Parquet, Avro and CSV.

Eh, I don't think that's the problem. If that was the problem, there are a zillion ways to chunk files; .tar is probably the most ubiquitous but there are others.

The bigger problem is that Protobuf is way harder to use. Part of the reason CSV is underspecified is it's simple enough it feels, at first glance, like it doesn't need specification. Protobuf has enough dark corners that I definitely don't know all of it, despite having used it pretty extensively.

I think Unicode Separated Values (USV) is a much better alternative and as another poster mentioned, I hope it takes off.

Re: Friends don't let friends export to CSV

#147

Post by bongodongobob » Tue, Mar 26, 2024, 2:04 AM UTC

The author seems to be missing the point of CSVs entirely. I looked him up expecting a fresh college grad, but am surprised to see he's probably in his early 30s. Seems to be in a dev bubble that doesn't actually work with users.

Try telling 45 year old salesman he needs to export his data in parquet. "Why would I need to translate it to French??"

I feel like I'm pretty up to date on stuff, and I've never heard of parquet or seen in as an option, in any software, ever.

Re: Friends don't let friends export to CSV

#148

Post by lostlogin » Tue, Mar 26, 2024, 2:05 AM UTC

ryncewynd wrote: post #116

hn_throwaway_99 wrote: post #111

Earlier quoted context omitted.

Totally agree. His arguments are basically "performance!" (which is honestly not important to 99% of CSV export users) and "It's underderspecified!" And while I can agree with the second, at least partly, in the real world the spec is essentially "Can you import it to Excel?". I'm amazed at how much programmers can discount "It already works pretty much everywhere" for the sake of more esoteric improvements. All that…

Are there unicode characters specifically for delimiters? If Excel had a standardised "Save as USV" option it would solve so many issues for me. I get so many broken CSVs from third-parties

> Are there unicode characters specifically for delimiters?

We could use the HL7 pipe ‘|’ and all enjoy that hell.

Re: Friends don't let friends export to CSV

#149

Post by wizzwizz4 » Tue, Mar 26, 2024, 2:06 AM UTC

jjgreen wrote: post #112

Cheer2171 wrote: post #77

Earlier quoted context omitted.

What do you mean there is no parquet support in Debian? Data formats should be supported in userspace and there are plenty of parquet libraries and userspace tools an apt-get away. There is exactly as much support for tar in Debian as there is for parquet.

You have searched for packages that names contain parquet in suite(s) bookworm, all sections, and all architectures. Sorry, your search gave no results https://packages.debian.org/search?suite=bookworm&searchon=n...

By the same logic, there's no Photoshop Document support – but GIMP and Krita both support it.

Re: Friends don't let friends export to CSV

#150

Post by bongodongobob » Tue, Mar 26, 2024, 2:10 AM UTC

392 wrote: post #123

I must have missed when Excel added Parquet support.

Yeah, I think the use case for CSVs 99% of the time is so you can load it into Excel and do things with it. Any other use case is going to involve a DB and if you're exporting databases to import to Excel or another DB, you are in fact doing it wrong.