Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Friends don't let friends export to CSV

kaveland.no

Post reply on HN
Print view View original

231–240 of 459 posts

Re: Friends don't let friends export to CSV

#231

Post by gxs »

mrgoldenbrown wrote: post #215
gxs wrote: post #161

Earlier quoted context omitted.

Agree. Not saying csv doesn’t have its issues, but I don’t think the author made a convincing argument. A lot of the issues the author brought up didn’t sound that bad and/or it sounds like he never looked at the source data first. If you’re doing work with large datasets, I think it’s a good practice to at least go and look at the source data briefly to see what to expect. This will give you a good idea of the forma…

IME most people don't know that using Excel to open and save a csv will silently mangle data. In our application leading zeros are significant, so we constantly get screwed by people trying to do quick manual edits and breaking the data. If we're lucky it breaks so badly the import fails. It's worse when the mangling results in structurally valid but wrong data.

I think what you’re saying is accurate, but it’s also important to be practical about stuff.

These are pretty well know excel limitations by now.

And really, anyone using excel who is somehow not aware of that limitation is probably not someone yet experienced enough to be working on a larger and/or mission critical dataset to begin with.

Are there exceptions? Sure. You might be tempted to cite the example of the incident where this happened to some biologists not too long ago, but mistakes happen. I’ve seen people make mistakes building android or iPhone using the right (TM) tools.

What is the exact number of mistakes where you make the decision to jump to a new format?

I’m not sure. This does happen eventually, but the author didn’t make a strong case here imo.

Re: Friends don't let friends export to CSV

#232

Post by lolive »

As a data architect in a big company, I cannot tell how harmful such a stupid data format CSV can be. All the possible semantics of the data has to be offloaded to either the brain of people [don’t do that! Just don’t!] or out-of-sync specs [better hidden in the CMS of the company that the Ark of Alliance, and outdated anyway] or obscure code or SQL queries [an opportunity for hilarious reverse engineering sessions, where you hate a retired developper forever for all the tricks he added inside code to circumvent poorly defined data. Then got away to Florida beach after hiring you.]

Re: Friends don't let friends export to CSV

#233

Post by lolive »

lolive wrote: post #232

As a data architect in a big company, I cannot tell how harmful such a stupid data format CSV can be. All the possible semantics of the data has to be offloaded to either the brain of people [don’t do that! Just don’t!] or out-of-sync specs [better hidden in the CMS of the company that the Ark of Alliance, and outdated anyway] or obscure code or SQL queries [an opportunity for hilarious reverse engineering sessions,…

The best thing I see really often is people sending the data model of a CSV file as, #guessWhat, ANOTHER CSV file!!!

[please kill me!]

Re: Friends don't let friends export to CSV

#234

Post by goatlover »

diogocp wrote: post #129
hn_throwaway_99 wrote: post #111

Earlier quoted context omitted.

Totally agree. His arguments are basically "performance!" (which is honestly not important to 99% of CSV export users) and "It's underderspecified!" And while I can agree with the second, at least partly, in the real world the spec is essentially "Can you import it to Excel?". I'm amazed at how much programmers can discount "It already works pretty much everywhere" for the sake of more esoteric improvements. All that…

> in the real world the spec is essentially "Can you import it to Excel?" And the answer to that is always no. You will it think it's yes because it works for you, but when you send it to someone who has a different Excel version or simply different regional settings, it won't work. The recipient will first have to figure out what dialect you used to export.

If the answer was always no, importing CSVs to Excel wouldn't be an expectation or widely used.

Re: Friends don't let friends export to CSV

#235

Post by josefx »

djbusby wrote: post #219
hn_throwaway_99 wrote: post #111

Earlier quoted context omitted.

Totally agree. His arguments are basically "performance!" (which is honestly not important to 99% of CSV export users) and "It's underderspecified!" And while I can agree with the second, at least partly, in the real world the spec is essentially "Can you import it to Excel?". I'm amazed at how much programmers can discount "It already works pretty much everywhere" for the sake of more esoteric improvements. All that…

ASCII has had field and record separators since like, forever. Wish we had kept using those.

Does it have nesting operators? I want to embedd ASCII within my ASCII fields. So I can have a table within my table.

Re: Friends don't let friends export to CSV

#236

Post by eviks »

outop wrote: post #216
nmz wrote: post #74

Earlier quoted context omitted.

oh boy. here's where it breaks - supporting "" (single) - Supporting newlines in "", oops, now you can't getline() and instead need to getdelim() - Supporting comments # (why is this even a thing) - Supporting multiple "" in a field - Escaping " with "" or \" - length based csv, so all fields are seekable. It's a mess, which one's your csv?

The vast majority of CSVs do not have strings which include either quotes or newlines. No CSV I have ever encountered has comments.

So you're fine with a lot of bugs in the case of a vast minority?

Re: Friends don't let friends export to CSV

#237

Post by lolive »

gerdesj wrote: post #117
GuB-42 wrote: post #97

As a French, there is another problem with CSV. In the French locale, the decimal point is the comma, so "121.5" is written "121,5". It means, of course, that the comma can't be used as a separator, so the semicolon is used instead. It means that depending whether or not the tool that exports the CSV is localized or not, you get commas or you get semicolons. If you are lucky, the tool that imports it speaks the same…

If only a totally separate data field separator character had been invented early on and been given its own key on the keyboard, coloured: "only use for field delimiting". You know as well as I do that it would have been co-opted into another rôle within weeks, probably as a currency indicator. You should probably use PSV - Point Separated Variable! Obviously we would need to adjust PSV to account for correct French…

Don’t look at the ASCII table, at entries 28 to 31. You are not mentally ready for what you will find there.

Re: Friends don't let friends export to CSV

#239

Post by Stranger43 »

nrdvana wrote: post #195
lmm wrote: post #188

Earlier quoted context omitted.

> Esperanto is a superior language to English. Not really. You can say it's more regular but that's because it sees barely any actual use; if it ever gained popularity it wouldn't stay regular. (And given that pilots speak in set phrases anyway, irregularity isn't really an issue). It's not a great language by any stretch, it's an awkward mismash of four european languages; sure it sounds kind of nice in an Italianat…

It would stay regular if there was a strict governing body for it that wasn't a Webster-style "whatever people are speaking is the new definition of correct". English really is a disaster of a language. There was a(nother) great XKCD about it just a few days ago. https://xkcd.com/2907/

Correction it's a disaster of at least 4 languages and this is probably why English is so hard to dethrone as it have no strict ownership so everyone is kind of equal in speaking it incorrectly.

Sometimes lack of rigidity is actually an feature that allows for things to sort of work that would be politically impossible if thinks had to be specified formally before being used.

Re: Friends don't let friends export to CSV

#240

Post by lolive »

josefx wrote: post #235
djbusby wrote: post #219

Earlier quoted context omitted.

ASCII has had field and record separators since like, forever. Wish we had kept using those.

Does it have nesting operators? I want to embedd ASCII within my ASCII fields. So I can have a table within my table.

A manager: I wish I could have a CSV inside my CSV.

Any sane person: NO!

Post reply on HN