Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Friends don't let friends export to CSV

kaveland.no

Post reply on HN
Print view View original

131–140 of 459 posts

Re: Friends don't let friends export to CSV

#131

Post by quesera »

chasil wrote: post #72
anymouse123456 wrote: post #38

Earlier quoted context omitted.

This is the biggest win IME. You have a (usually) portable transport format that can get the information into and out of an enormous variety of tools that do not necessarily require a software engineer in the middle. I'm also struggling with such a quick dismissal of human readable formats. It's a huge feature. What happens when there's a problem with a single CSV file in some pipeline that's been happily running fin…

In a POSIX shell, I actually prefer to use the bell character for IFS. while IFS="$(printf \\a)" read -r field1 field2... do ... done This works just as well as anything outside the range of printing characters. Getting records that contain newlines would be a bit trickier.

Heaven help you if you cat the file in a shell, though!

Re: Friends don't let friends export to CSV

#132

Post by quesera »

Every single use I've ever seen of CSV would be improved by the very simple change to TSV.

Even Excel can handle it.

It is far safer to munge data containing tabs (convert to spaces, etc), than commas (remove? convert to dots? escape?).

The better answer is to use ASCII separators as Lyndon Johnson intended, but that turns out to be asking a lot of data producers. Generating TSV is usually easier than generating CSV.

Re: Friends don't let friends export to CSV

#134

Post by nradov »

senknvd wrote: post #110
nradov wrote: post #66

Earlier quoted context omitted.

If both parties implement RFC 4180 and use a consistent character set encoding then I don't think there are actually any undefined behaviors. But in practice a lot of implementations are simply broken, including those from major tech companies that ought to know better.

I don't think RFC 4180 differentiates between an empty string and a null value. As long as you add a check that all string columns are free of empty values before writing you should be good. I think in polars it's df.filter(pl.col(pl.Utf8).str.len_bytes() == 0).shape[0] == 0 although there's probably a better way to write this.

Well I would consider differentiation between empty string versus null as simply being out of scope for CSV rather than undefined behavior. It was never intended as a complete database dump format.

Re: Friends don't let friends export to CSV

#135

Post by forgetfreeman »

ravenstine wrote: post #83
VHRanger wrote: post #81

Earlier quoted context omitted.

If you send someone JSON there's no guarantee the data is tabular, or even formatted, though

Better something that can be parsed than not parsable at all?

If you require parse logic to import even the most trivial data export you've failed at several tasks concurrently.

Re: Friends don't let friends export to CSV

#136

Post by eyegor »

layer8 wrote: post #106
GuB-42 wrote: post #97

As a French, there is another problem with CSV. In the French locale, the decimal point is the comma, so "121.5" is written "121,5". It means, of course, that the comma can't be used as a separator, so the semicolon is used instead. It means that depending whether or not the tool that exports the CSV is localized or not, you get commas or you get semicolons. If you are lucky, the tool that imports it speaks the same…

There are tools to convert between the formats. Either you have a defined data pipeline where you know what you get at each step and apply the necessary transformations. Or you get random files and, yes, have to inspect them and see how to convert them if necessary. It’s unfortunate that there isn’t a single CSV format, but for historical reasons it is what it is. It’s effectively more like a family of formats that s…

What kind of voodoo is this? I've always wanted something like this for non technical coworkers across countries but I didn't know it existed. I always just exported TSV and provided steps for how to import it (although I think most excel versions have native tsv support).

Re: Friends don't let friends export to CSV

#137

Post by rovr138 »

msla wrote: post #122
breadwinner wrote: post #98

The reason CSV is popular is because it is (1) super simple, and (2) the simplicity leads to ubiquity. It is extremely easy to add CSV export and import capability to a data tool, and that has come to mean that there are no data tools that don't support CSV format. Parquet is the opposite of simple. Even when good libraries are available (which it usually isn't), it is painful to read a Parquet file. Try reading a Pa…

> there are no data tools that don't support CSV format. They support CSV but not your CSV. For example, how does quoting work? Does quoting work?

I was working with a vendor’s csv recently…

They had never had a customer do X on Y field, so they never quoted it nor added code to quote it if needed..

Of course, we did X in one entry. Took me too long to find that which obviously messed up everything after.

Re: Friends don't let friends export to CSV

#138

Post by SPBS »

1. CSV is for ensuring compatibility with the widest range of consumers, not for ensuring best read or storage performance for consumers. (It is already more efficient than JSON because it can be streamed, and takes up less space than a JSON array of objects)

2. The only data type in CSV is a string. There is no null, there are no numbers. Anything else must be agreed upon between producer and consumer (or more commonly, a consumer looks at the CSV and decides how the producer formatted it). JSON also doesn’t include dates, you’re not going to see people start sending API responses as Apache Parquet. CSV is fiiine.

Re: Friends don't let friends export to CSV

#139

Post by thayne »

nradov wrote: post #66
aronhegedus wrote: post #6

My takeaway is that csv has some undefined behaviours, and it takes up space. I like that everyone knows about .csv files, and it's also completely human readable. So for <100mb I would still use csv.

If both parties implement RFC 4180 and use a consistent character set encoding then I don't think there are actually any undefined behaviors. But in practice a lot of implementations are simply broken, including those from major tech companies that ought to know better.

And the application doesn't try to convert the cells into non-string data types like numbers, dates, etc.

Re: Friends don't let friends export to CSV

#140

Post by chaps »

diogocp wrote: post #129
hn_throwaway_99 wrote: post #111

Earlier quoted context omitted.

Totally agree. His arguments are basically "performance!" (which is honestly not important to 99% of CSV export users) and "It's underderspecified!" And while I can agree with the second, at least partly, in the real world the spec is essentially "Can you import it to Excel?". I'm amazed at how much programmers can discount "It already works pretty much everywhere" for the sake of more esoteric improvements. All that…

> in the real world the spec is essentially "Can you import it to Excel?" And the answer to that is always no. You will it think it's yes because it works for you, but when you send it to someone who has a different Excel version or simply different regional settings, it won't work. The recipient will first have to figure out what dialect you used to export.

"Always no"? What. Really not sure what you mean here when you agree that the possibility of it working exists.

I get the sentiment -- when I request data through FOIA, I will almost always request it as "an excel format" because I know that I'll at least be able to import it. CSV is much less of a guarantee and will have issues -- missing quotes, wrong delimiters, inconsistent column counts, things like that. So requesting "an excel format" implies "make the best effort to give me something that will load in my excel, but without asking what version of excel I have". Removes a fair amount of hassle, especially when it took months to get the data. It also means that if they fuck up the columns by doing a conversion, you have some means of telling them that the data is simply wrong, rather than the data is hard to work with. It does mean dealing with [0-9]+GB sized excel files sometimes, though.

That all said, I prefer to share CSV files. Haven't had much of a problem with it and I can guarantee some consistency from my side. Like, the people I share files with aren't going to know what the hell a parquet file is. A CSV though? Just double click it if you're less technical, or open it up in less if you can use a terminal. It usually compresses well, despite what the author wrote.

Post reply on HN