Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Friends don't let friends export to CSV

kaveland.no

Post reply on HN
Print view View original

211–220 of 459 posts

Re: Friends don't let friends export to CSV

#211

Post by Dalewyn »

deathanatos wrote: post #181
Dalewyn wrote: post #154

Earlier quoted context omitted.

>I do hope "Unicode Separated Values" takes off. It's essentially just a slight tweak to CSV where the delimiters are special unicode characters Commas can be typed by anyone on any keyboard and readable by anyone. Special Unicode Characters(tm) can't be typed by anyone on any keyboard and readable by noone. Convenience is a virtue.

> Special Unicode Characters(tm) can't be typed by anyone on any keyboard and readable by noone. While I'm not a fan of USV, I do believe it is type-able on both macOS and Linux¹. The IME's character picker in both OSes contains all of the necessary characters, I think. (I use "␤" in commit messages, occasionally. That was a typed ␤, not copy/pasted from a Google query or such.) It would be quite tedious , I do admit…

Any character within reason can certainly be entered by way of Character Map in Windows or its equivalent in Linux or MacOS, but if you're arguing that then you don't understand the crux of my argument: Convenience is a virtue.

There is value in the delimiter simply being a key on practically any keyboard in existence. Anything that involves something more complicated than just pushing a single button on a keyboard (this includes IMEs) is a non-starter, because convenience is a virtue.

Re: Friends don't let friends export to CSV

#212

Post by quesera »

croes wrote: post #186
quesera wrote: post #132

Every single use I've ever seen of CSV would be improved by the very simple change to TSV. Even Excel can handle it. It is far safer to munge data containing tabs (convert to spaces, etc), than commas (remove? convert to dots? escape?). The better answer is to use ASCII separators as Lyndon Johnson intended, but that turns out to be asking a lot of data producers. Generating TSV is usually easier than generating CSV.

Don't change the data by replacing the separator. If the data contains a comma, tab etc. I want to receive a comma, tab etc. With proper escaping the separator doesn't matter. After that csv generating is pretty easy.

Right, the problem with CSV is that it's often generated incorrectly.

Commas do not need to be escaped inside quoted blocks. But now quoted blocks are not literal data, and you need to drop the enclosing quotes. And escape quotes inside them, or outside if the quote is literal. Also quotes must be balanced inside a delimited field. Etc.

Commas are just too common of a character to be a good delimiter.

Tabs are an improvement, because usually tabs are just whitespace, which is usually arbitrary, and never needs to be balanced.

But tabs are not perfect either. Just a vast improvement for most data, and a functional compromise for humans.

Re: Friends don't let friends export to CSV

#213

Post by mlhpdx »

I’ve always liked CSV. It’s a streaming friendly format so:

- the sender can produce it incrementally

- the receiver can begin processing it as soon as the first byte arrives (or, more roughly, unescaped newline)

- gzip compression works without breaking the streaming nature

Yeah, it’s a flawed interchange format. But in a closed system over HTTP it’s brilliant.

Re: Friends don't let friends export to CSV

#215

Post by mrgoldenbrown »

gxs wrote: post #161
prepend wrote: post #85

This article seems written by someone who never had to work with diverse data pipelines. I work with large volumes of data from many different sources. I’m lucky to get them to send csv. Of course there are better formats, but all these sources aren’t able to agree on some successful format. Csv that’s zipped is producible and readable by everyone. And that makes is more efficient. I’ve been reading these “everyone i…

Agree. Not saying csv doesn’t have its issues, but I don’t think the author made a convincing argument. A lot of the issues the author brought up didn’t sound that bad and/or it sounds like he never looked at the source data first. If you’re doing work with large datasets, I think it’s a good practice to at least go and look at the source data briefly to see what to expect. This will give you a good idea of the forma…

IME most people don't know that using Excel to open and save a csv will silently mangle data. In our application leading zeros are significant, so we constantly get screwed by people trying to do quick manual edits and breaking the data. If we're lucky it breaks so badly the import fails. It's worse when the mangling results in structurally valid but wrong data.

Re: Friends don't let friends export to CSV

#216

Post by outop »

nmz wrote: post #74
johnea wrote: post #64

I've written CSV exports in C from scratch, no external dependencies required. It's "Comma Separated Variables", it doesn't really need anymore specification than that. These files have always imported into M$ and libre office suites without issue.

oh boy. here's where it breaks - supporting "" (single) - Supporting newlines in "", oops, now you can't getline() and instead need to getdelim() - Supporting comments # (why is this even a thing) - Supporting multiple "" in a field - Escaping " with "" or \" - length based csv, so all fields are seekable. It's a mess, which one's your csv?

The vast majority of CSVs do not have strings which include either quotes or newlines.

No CSV I have ever encountered has comments.

Re: Friends don't let friends export to CSV

#217

Post by quesera »

chirau wrote: post #182
quesera wrote: post #132

Every single use I've ever seen of CSV would be improved by the very simple change to TSV. Even Excel can handle it. It is far safer to munge data containing tabs (convert to spaces, etc), than commas (remove? convert to dots? escape?). The better answer is to use ASCII separators as Lyndon Johnson intended, but that turns out to be asking a lot of data producers. Generating TSV is usually easier than generating CSV.

you are assuming the regular end user knows the difference between 4 spaces and a tab and the nuances that come with them trying to replace one with the other or why space between two values is different at one point from the next. Commas are, by far, better delimeters than tabs in the grand scheme of things and with both expert and regular users considered.

I disagree. The end user doesn't need to know the difference between 4 spaces and a tab. Tabs are just whitespace.

Tabs are uncommon but convenient whitespace. Commas are extremely common content. Tabs are a vastly better delimiter.

If you are wrapping source code in a CSV, a) you're doing it wrong, and b) you'll get bitten by newlines just as quickly! If you're including content that requires specific whitespace preservation, just escape the (usually rare) tabs.

TSV certainly is not perfect. But it solves the major problems for 95% of CSVs, and it's just as convenient for humans.

I do agree that one should not arbitrarily munge content. But note that HTML does munge whitespace, and we've never suffered meaningfully for it.

Re: Friends don't let friends export to CSV

#219

Post by djbusby »

hn_throwaway_99 wrote: post #111
prepend wrote: post #85

This article seems written by someone who never had to work with diverse data pipelines. I work with large volumes of data from many different sources. I’m lucky to get them to send csv. Of course there are better formats, but all these sources aren’t able to agree on some successful format. Csv that’s zipped is producible and readable by everyone. And that makes is more efficient. I’ve been reading these “everyone i…

Totally agree. His arguments are basically "performance!" (which is honestly not important to 99% of CSV export users) and "It's underderspecified!" And while I can agree with the second, at least partly, in the real world the spec is essentially "Can you import it to Excel?". I'm amazed at how much programmers can discount "It already works pretty much everywhere" for the sake of more esoteric improvements. All that…

ASCII has had field and record separators since like, forever. Wish we had kept using those.

Re: Friends don't let friends export to CSV

#220

Post by mrgoldenbrown »

danirod wrote: post #41

Friends don't let friends export to CSV -- in the data science field. But outside the data science field, my experience working on software programming these years is that it won't matter how beautiful your backoffice dashboards and web apps are, many non-technical business users will demand at some point CSV import and/or export capabilities, because it is easier for them to just dump all the data on a system into E…

But why the half measure of csv when it's just as simple to use a library to export to an actual excel file (which is really just xml) , which will properly preserve your data and make the business users happy.
Post reply on HN