Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Friends don't let friends export to CSV

kaveland.no

Post reply on HN
Print view View original

191–200 of 459 posts

Re: Friends don't let friends export to CSV

#191

Post by croes »

barrkel wrote: post #20
rietta wrote: post #5

Or export to CSV correctly and test with Excel and/or LibreOffice. Honestly CSV is a very simple, well defined format, that is decades old and is “obvious”. I’ve had far more trouble with various export to excel functions over the years, that have much more complex third-party dependencies to function. Parsing CSV correctly is not hard, you just can’t use split and be done with it. This has been my coding kata in eve…

CSV is not well-defined. Data in the wild doesn't even agree that it's comma separated. String encoding? Dates? Formatted numbers? Booleans (T/F/Y/N/etc)? Nested quotes? Nested CSV!? How about intermediate systems that muck things up. String encoding going through a pipeline with a misconfiguration in the middle. Data with US dates pasted into UK Excel and converted back into CSV, so that the data is a mix of m/d/yy…

Seems more like a problem of the exporting and importing software and not the format.

CSV gives you the freedom of choice how to write the data.

You need the french data format? You can write the french date format. You need # as separator? You can use # as separator.

Sure you can't read any file without knowing it's specification but that's not the usecase of CSV.

CSV is for data transfer between systems which know each other.

Re: Friends don't let friends export to CSV

#192

Post by duped »

croes wrote: post #189

>Numerical columns may also be ambigious, there's no way to know if you can read a numerical column into an integral data type, or if you need to reach for a float without first reading all the records. Most of the time you know the source pretty well and can simply ask about the value range.

If you just assume f64 then you have 53 bits of integer precision which is more than enough for the fast majority of applications. If JS hasn't proven this thoroughly, I don't know what has.

Obviously there are edges, but they're edges by nature. And like you say, you usually know the source pretty well.

Re: Friends don't let friends export to CSV

#194

Post by bee_rider »

deathanatos wrote: post #181
Dalewyn wrote: post #154

Earlier quoted context omitted.

>I do hope "Unicode Separated Values" takes off. It's essentially just a slight tweak to CSV where the delimiters are special unicode characters Commas can be typed by anyone on any keyboard and readable by anyone. Special Unicode Characters(tm) can't be typed by anyone on any keyboard and readable by noone. Convenience is a virtue.

> Special Unicode Characters(tm) can't be typed by anyone on any keyboard and readable by noone. While I'm not a fan of USV, I do believe it is type-able on both macOS and Linux¹. The IME's character picker in both OSes contains all of the necessary characters, I think. (I use "␤" in commit messages, occasionally. That was a typed ␤, not copy/pasted from a Google query or such.) It would be quite tedious , I do admit…

Would it be possible to just type the file using commas, semicolons, or pipes or something (whatever you happen to know you don’t have in your file) and then convert them using sed?

Re: Friends don't let friends export to CSV

#195

Post by nrdvana »

lmm wrote: post #188
prepend wrote: post #167

Earlier quoted context omitted.

I got a little snarky but I think the analogy holds. Esperanto is a superior language to English. And English has many flaws. Theoretically it would be better to have all pilots and airports learn an efficient language. But it would be stupid and immature to seriously write a blog post about that, especially without talking about all the flaws in that plan.

> Esperanto is a superior language to English. Not really. You can say it's more regular but that's because it sees barely any actual use; if it ever gained popularity it wouldn't stay regular. (And given that pilots speak in set phrases anyway, irregularity isn't really an issue). It's not a great language by any stretch, it's an awkward mismash of four european languages; sure it sounds kind of nice in an Italianat…

It would stay regular if there was a strict governing body for it that wasn't a Webster-style "whatever people are speaking is the new definition of correct".

English really is a disaster of a language. There was a(nother) great XKCD about it just a few days ago. https://xkcd.com/2907/

Re: Friends don't let friends export to CSV

#196

Post by nrdvana »

prepend wrote: post #169
msla wrote: post #118

Earlier quoted context omitted.

> Csv that’s zipped is producible and readable by everyone. And that makes is more efficient. If only CSV were CSV, as opposed to some form that's 80-90% CSV by line count with enough oddities to really make the parser ugly and hard to read. See, the sweet spot isn't something completely unstructured, because then you feel justified in throwing up (your hands) and declaring defeat. The sweet spot is a file that's suf…

There’s definitely hair to deal with and it’s a little messy, but it’s never a blocker. But I’d say the error rate is actually very low, maybe .1-1% and nowhere near 10-20% of data being messed up.

Depends on your tools, I suppose. I'd just like to share this:

https://metacpan.org/pod/Data::TableReader::Decoder::IdiotCS...

Re: Friends don't let friends export to CSV

#197

Post by arp242 »

breadwinner wrote: post #98

The reason CSV is popular is because it is (1) super simple, and (2) the simplicity leads to ubiquity. It is extremely easy to add CSV export and import capability to a data tool, and that has come to mean that there are no data tools that don't support CSV format. Parquet is the opposite of simple. Even when good libraries are available (which it usually isn't), it is painful to read a Parquet file. Try reading a Pa…

> Parquet is the opposite of simple. Even when good libraries are available (which it usually isn't), it is painful to read a Parquet file. Try reading a Parquet file using Java and Apache Parquet lib, for example.

I skimmed their docs a bit: https://parquet.apache.org/docs/

I would not look forward to implementing that.

It all seems rather complex, and even worse: not actually all that well described. I suppose all the information is technically there, but it's really not a well-design well-written specification that's easy to implement. The documentation seems like an afterthought.

This is probably why good libraries are rare.

Re: Friends don't let friends export to CSV

#198

Post by hackerlight »

prepend wrote: post #85

This article seems written by someone who never had to work with diverse data pipelines. I work with large volumes of data from many different sources. I’m lucky to get them to send csv. Of course there are better formats, but all these sources aren’t able to agree on some successful format. Csv that’s zipped is producible and readable by everyone. And that makes is more efficient. I’ve been reading these “everyone i…

It's a premature optimization issue. If you don't have special requirements like IO throughput, or mission critical data accuracy guarantees, be biased towards picking the format that anyone can easily open in a spreadsheet.

Re: Friends don't let friends export to CSV

#199

Post by potatoicecoffee »

fbdab103 wrote: post #170
ryncewynd wrote: post #116

Earlier quoted context omitted.

Are there unicode characters specifically for delimiters? If Excel had a standardised "Save as USV" option it would solve so many issues for me. I get so many broken CSVs from third-parties

My significantly bigger beef would be all of the auto-formatting Excel does to mangle data. Excel loves to turn entries into dates. Human genes had to be renamed so as to avoid this Excel features.

excel now has a prompt so you can tell it to not convert stuff automatically

Re: Friends don't let friends export to CSV

#200

Post by fbdab103 »

potatoicecoffee wrote: post #199
fbdab103 wrote: post #170

Earlier quoted context omitted.

My significantly bigger beef would be all of the auto-formatting Excel does to mangle data. Excel loves to turn entries into dates. Human genes had to be renamed so as to avoid this Excel features.

excel now has a prompt so you can tell it to not convert stuff automatically

Gasp. Big news. I do not recall ever seeing this, so I wonder if $JOB is running some hilariously outdated version for compatibility with a load bearing VBA script.
Post reply on HN