Friends don't let friends export to CSV • Hacker News Board

Re: Friends don't let friends export to CSV

#91

Post by 2devnull » Tue, Mar 26, 2024, 12:14 AM UTC

alserio wrote: post #61

Black616Angel wrote: post #18

I never liked articles about how you should replace CSV with some other format while pulling some absolutely idiotic reasons out of their rear... 1. CSV is underspecified Okay, so specify it for your use case and you're done? E.g use rfc3339 instead of the straw-man 1-1-1970 and define how no value looks like, which is mostly an empty string. 2. CSV files have terrible compression and performance Okay, who in their r…

I agree with your other points but the first point misses the mark. Even you specify a format, you cannot use the file for exporting data between systems and organizations if they don't all agree on that format. CSV does not have a reasonable way to encode that is using a specific spec. I can open your data with my tools and silently misinterpret it. But if you are only exporting data between yourself, that's another…

You can use excel as the lingua franca. Also give them a row/col counts. Most problems solved in two easy steps.

Re: Friends don't let friends export to CSV

#92

Post by onethumb » Tue, Mar 26, 2024, 12:15 AM UTC

CSV has some limits and difficulties, but has massive benefits in terms of readability, portability, etc.

I feel like USV (Unicode Separated Values) neatly improves CSV while maintaining most of its benefits.

https://github.com/sixarm/usv

Re: Friends don't let friends export to CSV

#93

Post by 2devnull » Tue, Mar 26, 2024, 12:16 AM UTC

bluenose69 wrote: post #24

benob wrote: post #15

Can Parquet be read/parsed in almost every programming language with very little effort?

I just tried it in R. The relevant package seems to be "arrow", so I did install.packages("arrow") and then I did ?read_parquet to get an example. I tried the example, and got the error message as follows. This sort of error is really quite uncommon in R. So my answer to the "with little effort" is "no", at least for R. > tf write_parquet(mtcars, tf) Error in parquet___WriterProperties___Builder__create() : Cannot ca…

Arrow going back and forth between r/python can be a catch too iirc.

Re: Friends don't let friends export to CSV

#94

Post by martinflack » Tue, Mar 26, 2024, 12:19 AM UTC

yetihehe wrote: post #21

Macha wrote: post #14

Earlier quoted context omitted.

> Parsing CSV correctly is not hard, you just can’t use split and be done with it. Parsing RFC-compliant CSVs and telling clients to go away with non-compliant CSVs is not hard. Parsing real world CSVs reliably is simply impossible. The best you can do is heuristics. How do you interpret this row of CSV data? 1,5,The quotation mark "" is used...,2021-1-1 What is the third column? The RFC says that it should just be l…

For added fun, last column should be 1-2-2021.

Oh that's easy, it's Janreburary Firscond 2021.

Re: Friends don't let friends export to CSV

#95

Post by efxhoy » Tue, Mar 26, 2024, 12:22 AM UTC

kembrek wrote: post #69

If one could export to Parquet from Microsoft Excel, I think this would be a goer. Until such time, it seems likely many will stick with CSVs.

v1 or v2?

Re: Friends don't let friends export to CSV

#96

Post by xupybd » Tue, Mar 26, 2024, 12:25 AM UTC

danirod wrote: post #41

Friends don't let friends export to CSV -- in the data science field. But outside the data science field, my experience working on software programming these years is that it won't matter how beautiful your backoffice dashboards and web apps are, many non-technical business users will demand at some point CSV import and/or export capabilities, because it is easier for them to just dump all the data on a system into E…

Exactly Excel is the UI they know. This trumps every technical argument you can come up with. People don't want to throw out 20 years of experience with a tool to use your custom UI.

Re: Friends don't let friends export to CSV

#97

Post by GuB-42 » Tue, Mar 26, 2024, 12:25 AM UTC

As a French, there is another problem with CSV.

In the French locale, the decimal point is the comma, so "121.5" is written "121,5". It means, of course, that the comma can't be used as a separator, so the semicolon is used instead.

It means that depending whether or not the tool that exports the CSV is localized or not, you get commas or you get semicolons. If you are lucky, the tool that imports it speaks the same language. If you are unlucky, it doesn't, but you can still convert it. If you are really unlucky, then you get commas for both decimal numbers and separators, making the file completely unusable.

There is a CSV standard, RFC 4180, but no one seems to care.

Re: Friends don't let friends export to CSV

#98

Post by breadwinner » Tue, Mar 26, 2024, 12:26 AM UTC

The reason CSV is popular is because it is (1) super simple, and (2) the simplicity leads to ubiquity. It is extremely easy to add CSV export and import capability to a data tool, and that has come to mean that there are no data tools that don't support CSV format.

Parquet is the opposite of simple. Even when good libraries are available (which it usually isn't), it is painful to read a Parquet file. Try reading a Parquet file using Java and Apache Parquet lib, for example.

Avro is similar. Last I checked there are two Avro libs for C# and each has its own issues.

Until there is a simple format that has ubiquitous libs in every language, CSV will continue to be the best format despite the issues caused by under-specification. Google Protobuf is a lot closer than Parquet or Avro. But Protobuf is not a splitable format, which means it is not Big Data friendly, unlike Parquet, Avro and CSV.

Re: Friends don't let friends export to CSV

#99

Post by gerdesj » Tue, Mar 26, 2024, 12:27 AM UTC

prepend wrote: post #85

This article seems written by someone who never had to work with diverse data pipelines. I work with large volumes of data from many different sources. I’m lucky to get them to send csv. Of course there are better formats, but all these sources aren’t able to agree on some successful format. Csv that’s zipped is producible and readable by everyone. And that makes is more efficient. I’ve been reading these “everyone i…

... And what's more, you'll be an Engineer my son.

Re: Friends don't let friends export to CSV

#100

Post by BenjiWiebe » Tue, Mar 26, 2024, 12:28 AM UTC

chasil wrote: post #72

anymouse123456 wrote: post #38

Earlier quoted context omitted.

This is the biggest win IME. You have a (usually) portable transport format that can get the information into and out of an enormous variety of tools that do not necessarily require a software engineer in the middle. I'm also struggling with such a quick dismissal of human readable formats. It's a huge feature. What happens when there's a problem with a single CSV file in some pipeline that's been happily running fin…

In a POSIX shell, I actually prefer to use the bell character for IFS. while IFS="$(printf \\a)" read -r field1 field2... do ... done This works just as well as anything outside the range of printing characters. Getting records that contain newlines would be a bit trickier.

I think IFS=$'\a' works too.