Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Friends don't let friends export to CSV

kaveland.no

Post reply on HN
Print view View original

311–320 of 459 posts

Re: Friends don't let friends export to CSV

#311

Post by geraldhh »

Stranger43 wrote: post #227
diogocp wrote: post #129

Earlier quoted context omitted.

> in the real world the spec is essentially "Can you import it to Excel?" And the answer to that is always no. You will it think it's yes because it works for you, but when you send it to someone who has a different Excel version or simply different regional settings, it won't work. The recipient will first have to figure out what dialect you used to export.

At this point i suspect excel is as dangerous as powerpoint to the quest for sharing and improving human knowledge, in that it gives the illusion of allowing people to analyze data but not the tools to make sure they get their data and model lined up correctly.

otoh it could be instrumental precisely because it is flexible and doesn't require a lot of forethought

Re: Friends don't let friends export to CSV

#312

Post by chongli »

chaps wrote: post #253
throwaway2037 wrote: post #205

Earlier quoted context omitted.

> when I request data through FOIA Fascinating. Can you share any details? Did you ever think to share some of your interesting finds here on HN as a submission?

Every now and then, yep :) https://mchap.io/that-time-the-city-of-seattle-accidentally-...

That was a really fascinating story! Thanks for sharing.

Re: Friends don't let friends export to CSV

#314

Post by eska »

It's strange to me that people complain about some variety in CSV files while acting as if parquet was one specific file format that's set in stone. They can't even decide which features are core, and the file format has many massive backwards-incompatible changes already. If you give me a parquet file I cannot guarantee that I can read it, and if I produce one I cannot guarantee that you can.

I treat formats such as parquet as I generally do: I try to allow various different inputs, and produce standard outputs. Parquet is something I allow purely as an optimization. CSV is the common default all of my tools have (UTF-8 without BOM, international locale, comma separator, quoting at the start of the value optional, standards-compliant date format or unix timestamps). Users generally don't have any issue with adapting their files to that format if there's any difference.

Re: Friends don't let friends export to CSV

#315

Post by kmmlng »

GuB-42 wrote: post #97

As a French, there is another problem with CSV. In the French locale, the decimal point is the comma, so "121.5" is written "121,5". It means, of course, that the comma can't be used as a separator, so the semicolon is used instead. It means that depending whether or not the tool that exports the CSV is localized or not, you get commas or you get semicolons. If you are lucky, the tool that imports it speaks the same…

In my experience, this is only a problem when you are using Excel. It's ridiculous how bad Excel is at handling CSVs, I really cannot comprehend it. If you use LibreOffice all your problems magically disappear.

Re: Friends don't let friends export to CSV

#316

Post by roelschroeven »

Freak_NL wrote: post #279
diogocp wrote: post #129

Earlier quoted context omitted.

> in the real world the spec is essentially "Can you import it to Excel?" And the answer to that is always no. You will it think it's yes because it works for you, but when you send it to someone who has a different Excel version or simply different regional settings, it won't work. The recipient will first have to figure out what dialect you used to export.

Oh absolutely. Don't forget about the giant mess Microsoft made in countries like the Netherlands, where Excel will refuse to automatically open comma-separated values files (you know, CSV), unless the separator is a semicolon — because someone in the past thought that was how the Dutch did it. You want people to be able to open your nice and sensible CSV files in Excel? First figure out which arcane locale specific…

Many countries use the comma as decimal separator, and Microsoft in its infinite wisdom thinks that data interchange formats should follow regional formatting settings (that's unbelievably stupid; I'll never understand how such an enormous error not only came to be, but was never corrected). That makes the comma unusable as column separator in those countries for exchange of numerical data.

Re: Friends don't let friends export to CSV

#318

Post by kibwen »

goatlover wrote: post #234
diogocp wrote: post #129

Earlier quoted context omitted.

> in the real world the spec is essentially "Can you import it to Excel?" And the answer to that is always no. You will it think it's yes because it works for you, but when you send it to someone who has a different Excel version or simply different regional settings, it won't work. The recipient will first have to figure out what dialect you used to export.

If the answer was always no, importing CSVs to Excel wouldn't be an expectation or widely used.

The answer is "always no" because the question is inherently underspecified, precisely because importing things into Excel is more complicated and context-dependent than it appears on the surface.

Re: Friends don't let friends export to CSV

#319

Post by ranguna »

elric wrote: post #304
ric2b wrote: post #301

Earlier quoted context omitted.

Now all your numbers are strings

It's a text file. All your numbers were already strings. Nothing has changed.

There's a difference between "1" and 1. When you import a csv and try to do maths on a "number" you won't get the expected result. Some importers won't even allow you to specify that "number" columns are numbers, they'll outright fail and force you to say it's a string, or you'll have to specify which columns are "numbers" and map the strings to numbers on the importer side.

If they are numbers to begin with (not "numbers"), you can just import the csv and you'll get the expected result out of the box.

In the end things are just 1s and 0s, but that doesn't mean we only ever do binary operations on data at the abstraction layer we humans operate, so saying it's just 0s and 1s or just strings is not very smart.

Re: Friends don't let friends export to CSV

#320

Post by heresie-dabord »

al_borland wrote: post #203
layer8 wrote: post #106

Earlier quoted context omitted.

There are tools to convert between the formats. Either you have a defined data pipeline where you know what you get at each step and apply the necessary transformations. Or you get random files and, yes, have to inspect them and see how to convert them if necessary. It’s unfortunate that there isn’t a single CSV format, but for historical reasons it is what it is. It’s effectively more like a family of formats that s…

>sep=; Thank you! This is a game changer. As I was reading through these comments I was thinking how much better it would be if the separator could be specified in the file, but it would only be useful for my own stuff, if I was to do that on my own. I’ll be trying this first thing tomorrow at work. I don’t do as much with CSVs as I used to, but am currently working with them a lot, and have a single entry that alway…

I have sometimes distributed a CSV file with a similarly-named text file that contains key-value pairs to aid in the use of the CSV file.

A minimal text file of this type would contain:

    #key=value
    sep=;
Post reply on HN