Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

So You Want to Write Your Own CSV code

tburette.github.io

Post reply on HN
Print view View original

31–40 of 127 posts

Re: So You Want to Write Your Own CSV code

#31

Post by wiredfool »

keeperofdakeys wrote: post #21
mantrax5 wrote: post #6

Why are people using CSV when better (and less fuzzily defined) solutions exist, such as JSON?

If you have simple data, why use something as complicated as JSON? For a recent project, I had a simple CSV file with an int and float per row; using JSON would probably double the datasize. I used a simple string.split(",") for the javascript decoder, because I control the data, and know it's safe. I don't need another javascript library (I'd probably do differently if I had a standard library, not a hodge-podge of…

Why not use both?

I've had to() parse json embedded in a field in a csv file. Unquoted of course.

Until I explained to the other developer just how stupid that was.

Re: So You Want to Write Your Own CSV code

#32

Post by mrweasel »

tikumo wrote: post #27
mrweasel wrote: post #22

The most retard structure I've seen in a CSV file relates to the "What if the character separating fields is not a comma?". We get "CSV" files from Klarna, an invoicing company, with the payments they've processed for us. Because we're Danish and they are Swedish, it's not really weird that they would use comma as the decimal separator. So to compensate for having used the comma, they for some reason picks ", " ( tha…

it can be irritating, but you can just as easy parse ", " to "|" or something, by simple string replacing, pre parsing..

True, but in my mind picking ", " indicate to me that they don't care or don't know what they're doing. I often run into something similar with XML. I've had more than one partner call or write me saying that the elements in a file are not in the right order. Every single time they've admitted to not actually using an XML parser.

Don't do things that screw up the standard tools other developers depend on.

Re: So You Want to Write Your Own CSV code

#34

Post by lignuist »

sunir wrote: post #30
tikumo wrote: post #27

Earlier quoted context omitted.

it can be irritating, but you can just as easy parse ", " to "|" or something, by simple string replacing, pre parsing..

Think it through. What if there is free text in the field? "How are you, Sally?"

You can replace all commas with a placeholder (e.g. "#COMMA#"), replace the delimiter with a comma, parse the document and then replace all placeholders in the data with ",".

Re: So You Want to Write Your Own CSV code

#36

Post by slg »

CSV are a headache. Like the article says, RFC4180 doesn't necessarily represent the real world. However sometimes you just have to reject things that aren't spec.

Not too long ago I was struggling with one of these CSV issues and received some good advice from Hans Passant [1] on a Stack Overflow question pertaining to my problem (emphasis mine):

"It is pretty important that you don't try to fix it. That will make you responsible for bad data for a long time. Reject the file for being improperly formatted. If they hassle you about it then point out that it is not RFC-4180 compatible. There's another programmer somewhere that can easily fix this."

It makes perfect sense in hindsight. If you accept a malformed CSV file, people will expect you to accept any malformed data that has a CSV extension. You are taking on a lot of extra responsibility to cover for the lack of work by another programmer. Odds are they can make a change to fix the problem that takes a fraction of the time it would take you work around it. You just have to raise the issue.

I realize that rejecting bad files isn't really possible in every circumstance. But I have a feeling it is an option more times than you might initially think.

[1] - http://stackoverflow.com/users/17034/hans-passant

Re: So You Want to Write Your Own CSV code

#37

Post by lukeschlather »

jzwinck wrote: post #20
mantrax5 wrote: post #6

Why are people using CSV when better (and less fuzzily defined) solutions exist, such as JSON?

If your data are rectangular and you care about performance, CSV is better than JSON just because it avoids repetitive key names everywhere. Then again, if your data are rectangular and you really care about performance, you would not use any of these (you might use HDF5, which has support in many programming languages and will destroy the others in terms of speed).

JSON is almost a subset of CSV, with the understanding that you have to wrap every line in [], the document in [], and every field must be quoted. (And JSON doesn't have built-in support for headings, so you need to write a little loop instead of the library building a hash for you.)

So no, if you control input and output, JSON is still easier to use than CSV, and just as performant. JSON stores straight arrays just fine. It's not the format's fault so many people choose to store hashes with it.

Re: So You Want to Write Your Own CSV code

#38

Post by Moto7451 »

leni536 wrote: post #19
seanwoods wrote: post #16

This article makes it much more complicated than it needs to be. It tries to be all things to all people. In practice you're going to have to sacrifice some functionality for the sake of usability and your own sanity. When I add a CSV import feature to a project I'm working on, I tell people "this works with MS Excel flavor of CSV." This covers most, if not all, real world cases because in my world the people who wan…

Does it work with Hungarian Ms Excel? It uses semicolons as delimiters.

If all you care about is Excel compatibility you can add "sep=," on the first line. You can also use the Text Import Wizard. Changing the extension to .txt should cause Excel to show the Wizard upon opening the file.

Re: So You Want to Write Your Own CSV code

#39

Post by draegtun »

qnaal wrote: post #26
jstsch wrote: post #2

So, which library? CSV is a mess.

perl's Text::CSV http://search.cpan.org/~makamaka/Text-CSV-1.32/lib/Text/CSV....

Here's some links which (will always) point to latest versions on MetaCPAN:

http://p3rl.org/Text::CSV | http://p3rl.org/Text::CSV_XS

Post reply on HN