Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Friends don't let friends export to CSV

kaveland.no

Post reply on HN
Print view View original

201–210 of 459 posts

Re: Friends don't let friends export to CSV

#201

Post by al_borland »

deathanatos wrote: post #181
Dalewyn wrote: post #154

Earlier quoted context omitted.

>I do hope "Unicode Separated Values" takes off. It's essentially just a slight tweak to CSV where the delimiters are special unicode characters Commas can be typed by anyone on any keyboard and readable by anyone. Special Unicode Characters(tm) can't be typed by anyone on any keyboard and readable by noone. Convenience is a virtue.

> Special Unicode Characters(tm) can't be typed by anyone on any keyboard and readable by noone. While I'm not a fan of USV, I do believe it is type-able on both macOS and Linux¹. The IME's character picker in both OSes contains all of the necessary characters, I think. (I use "␤" in commit messages, occasionally. That was a typed ␤, not copy/pasted from a Google query or such.) It would be quite tedious , I do admit…

Even if the csv isn’t being typed out by hand, when importing into Excel and the delimiter needs to be manually entered, because it isn’t one of the common ones Excel has a radio button for… it is nice to be able to easily type it.

Re: Friends don't let friends export to CSV

#203

Post by al_borland »

layer8 wrote: post #106
GuB-42 wrote: post #97

As a French, there is another problem with CSV. In the French locale, the decimal point is the comma, so "121.5" is written "121,5". It means, of course, that the comma can't be used as a separator, so the semicolon is used instead. It means that depending whether or not the tool that exports the CSV is localized or not, you get commas or you get semicolons. If you are lucky, the tool that imports it speaks the same…

There are tools to convert between the formats. Either you have a defined data pipeline where you know what you get at each step and apply the necessary transformations. Or you get random files and, yes, have to inspect them and see how to convert them if necessary. It’s unfortunate that there isn’t a single CSV format, but for historical reasons it is what it is. It’s effectively more like a family of formats that s…

>sep=;

Thank you! This is a game changer. As I was reading through these comments I was thinking how much better it would be if the separator could be specified in the file, but it would only be useful for my own stuff, if I was to do that on my own.

I’ll be trying this first thing tomorrow at work. I don’t do as much with CSVs as I used to, but am currently working with them a lot, and have a single entry that always throws the resulting Excel file off, and I can’t be bothered to figure out how to escape it (I spent too much time in the past messing with that and got nowhere). This sep= lines will hopefully solve my issues.

Re: Friends don't let friends export to CSV

#204

Post by TimTheTinker »

rossdavidh wrote: post #174

"You give up human readable files, but what you gain in return is..." Stop right there. You lose more than you gain. Plus, taking the data out of [proprietary software app my client's data is in] in csv is usually easy. Taking the data out in Apache Parquet is...usually impossible, but if it is possible at all you'll need to write the code for it. Loading the data into [proprietary software app my client wants data p…

I think his arguments apply more closely to SQLite databases. They're not directly human readable, but boy are there a lot of tools for working with them.

Re: Friends don't let friends export to CSV

#205

Post by throwaway2037 »

chaps wrote: post #140
diogocp wrote: post #129

Earlier quoted context omitted.

> in the real world the spec is essentially "Can you import it to Excel?" And the answer to that is always no. You will it think it's yes because it works for you, but when you send it to someone who has a different Excel version or simply different regional settings, it won't work. The recipient will first have to figure out what dialect you used to export.

"Always no"? What. Really not sure what you mean here when you agree that the possibility of it working exists. I get the sentiment -- when I request data through FOIA, I will almost always request it as "an excel format" because I know that I'll at least be able to import it. CSV is much less of a guarantee and will have issues -- missing quotes, wrong delimiters, inconsistent column counts, things like that. So req…

   > when I request data through FOIA
Fascinating. Can you share any details? Did you ever think to share some of your interesting finds here on HN as a submission?

Re: Friends don't let friends export to CSV

#206

Post by throwaway2037 »

jgord wrote: post #165
whakim wrote: post #130

Earlier quoted context omitted.

To me this criticism feels excessive. It feels like the author is describing their frustrations with internal usage of CSVs - there's no mention of customers and non-technical stakeholders at all. I think it goes without saying that Parquet files and other non-human-readable formats are a nonstarter when working with external stakeholders and the last paragraph makes that clear - if the end-user wants CSV, give them…

btw, xsv has solved most of my problems dealing with 'large' 40GB csv files

xsv? I never heard of it. This one? https://github.com/BurntSushi/xsv

If yes, looks very cool. Plus, bonus HN/Internet points for being written in Rust!

Re: Friends don't let friends export to CSV

#207

Post by micheljansen »

fshr wrote: post #142
sfRattan wrote: post #107

Earlier quoted context omitted.

Going by the Wikipedia article and included map, use of comma versus period as decimal separators is roughly an even split: https://en.wikipedia.org/wiki/Decimal_separator https://commons.wikimedia.org/wiki/File:DecimalSeparator.svg

There's definitely a big distribution disparity. 11 of the 15 most populous countries use the period for decimals.

Most of Europe is in the fat long tail though, as those countries are counted individually.

Re: Friends don't let friends export to CSV

#208

Post by munchler »

arp242 wrote: post #197
breadwinner wrote: post #98

The reason CSV is popular is because it is (1) super simple, and (2) the simplicity leads to ubiquity. It is extremely easy to add CSV export and import capability to a data tool, and that has come to mean that there are no data tools that don't support CSV format. Parquet is the opposite of simple. Even when good libraries are available (which it usually isn't), it is painful to read a Parquet file. Try reading a Pa…

> Parquet is the opposite of simple. Even when good libraries are available (which it usually isn't), it is painful to read a Parquet file. Try reading a Parquet file using Java and Apache Parquet lib, for example. I skimmed their docs a bit: https://parquet.apache.org/docs/ I would not look forward to implementing that. It all seems rather complex, and even worse: not actually all that well described. I suppose all…

Thanks for the link. I couldn't even get past this part:

> Apache Parquet is a columnar storage format available to any project in the Hadoop ecosystem

Nope and nope.

Re: Friends don't let friends export to CSV

#209

Post by palmfacehn »

hn_throwaway_99 wrote: post #111
prepend wrote: post #85

This article seems written by someone who never had to work with diverse data pipelines. I work with large volumes of data from many different sources. I’m lucky to get them to send csv. Of course there are better formats, but all these sources aren’t able to agree on some successful format. Csv that’s zipped is producible and readable by everyone. And that makes is more efficient. I’ve been reading these “everyone i…

Totally agree. His arguments are basically "performance!" (which is honestly not important to 99% of CSV export users) and "It's underderspecified!" And while I can agree with the second, at least partly, in the real world the spec is essentially "Can you import it to Excel?". I'm amazed at how much programmers can discount "It already works pretty much everywhere" for the sake of more esoteric improvements. All that…

JSON objects as a CSV field has been mostly agreeable for my usage. It would be nice if some of the spreadsheet apps displayed the object tree.

Re: Friends don't let friends export to CSV

#210

Post by throwaway2037 »

rgmerk wrote: post #71
fragmede wrote: post #59

Earlier quoted context omitted.

In case it wasn't clear, I want software engineers in the finance industry to implement sqlite import and export in their various pieces of software, not too give up on lucrative, existing contracts, obviously.

The inertia around CSV in the finance industry is incredibly strong, and let’s just say some of the biggest industry players aren’t exactly reknowned for their cutting-edge practices when it comes to data interchange formats.

For interchange between banks, XML is also very common, because it is usually accompanied with a schema doc.
Post reply on HN