Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

So You Want to Write Your Own CSV code

tburette.github.io

Post reply on HN
Print view View original

91–100 of 127 posts

Re: So You Want to Write Your Own CSV code

#91

Post by mschuster91 »

barrkel wrote: post #55
slg wrote: post #36

CSV are a headache. Like the article says, RFC4180 doesn't necessarily represent the real world. However sometimes you just have to reject things that aren't spec. Not too long ago I was struggling with one of these CSV issues and received some good advice from Hans Passant [1] on a Stack Overflow question pertaining to my problem (emphasis mine): "It is pretty important that you don't try to fix it. That will make y…

On the other hand, the ability to handle all kinds of input can be a chief selling point of your product. In my current job, the most common "invalid" CSV format we get is .xlsx files. So I wrote an .xlsx parser (way, way faster than Apache POI). Another interesting hiccup to consider is CSV inside individual fields - i.e. recursive CSV. There are various ways to handle this, but in my company's line of business the…

> Likely the next invalid format we'll have to parse is PDFs containing tables...

cough people doing e-invoicing with pdf's...

Re: So You Want to Write Your Own CSV code

#92

Post by encoderer »

Early on in my career, just a year out of school, I, for some absurd reason, had the idea to build my own date library.

Primarily, I didn't fully understand the date objects and functions available in the languages/libraries i was using so simple things like formatting a string date seemed difficult to me.

This was an awful idea. Dreadful.

I came up with all sort of delightful helper methods to cover common use cases like adding one month to the current date. I made this decision to represent dates internally with a timestamp, so adding a month is easy, right?! No. ...What's 1 month from January 31st? February 28th? Well then what's 1 month from February 28th? The list of edge cases goes on.

Most things in life are more complicated than they, at first, seem.

Re: So You Want to Write Your Own CSV code

#93

Post by mschuster91 »

I usually take advantage of the fixed formats of the individual exporting tool. Everyone does it a bit different - so what? I have a php parser for it and adapt it for every of my clients. It's cheaper to have a small parser, adapted for the customer's needs, than having one 10k SLOC library to handle a boatload of files...

Re: So You Want to Write Your Own CSV code

#94

Post by earino »

qnaal wrote: post #26
jstsch wrote: post #2

So, which library? CSV is a mess.

perl's Text::CSV http://search.cpan.org/~makamaka/Text-CSV-1.32/lib/Text/CSV....

I really have to second this. It's fast, it's smart, it handles almost anything, and it reliably gives good error messages. It's an important part of my data science toolbelt.

Re: So You Want to Write Your Own CSV code

#95

Post by voltagex_ »

michaelmior wrote: post #46

The best tool I've found for working with CSV files is csvkit[1]. I've run into some of the issues mentioned in the article and it's handled them all gracefully. It's basically a bunch of scripts mirroring sort, grep, cut, etc. but specifically for dealing with CSV files. [1] http://csvkit.readthedocs.org/

Hey, this looks good. I've also used csvfix [1] to get me out of trouble before.

1: http://neilb.bitbucket.org/csvfix/

Re: So You Want to Write Your Own CSV code

#96

Post by NaNaN »

Why CSV is not just for readability? I think RFC is sometimes too pedantic, that it let CSV can handle both plain text and binary files. COMMA is not just a COMMA, but a COMMA not in different environments. Why should we use the phrase CSV or Comma Separated Values just for RFC?

CSV or Comma Separated Values are not only for RFC, but also for EVERYONE who wants to use this word or phrase. Pedantry sucks!

Re: So You Want to Write Your Own CSV code

#97

Post by kemayo »

ohwaitnvm wrote: post #83
kemayo wrote: post #63

We actually use CSV-reading as an incidental part of a hiring exercise. We provide a really simple homemade CSV parser as part of a PHP project, with a "could you find and fix bugs in this?" instruction. The way to get full marks is to rip out the parser and replace it with the appropriate standard library function.

I like this. Only thing that I don't like is that many candidates will assume that they have to fix the code within the parser, given those instructions, even if they know that a battle-tested library is how they would actually do it. I hope you accept an off-hand comment such as, "ew, why is this hand-rolled" as a sufficient indicator in favor of your solution.

Such a comment would be acceptable, yeah. So long as we can tell they looked at it and thought "wow, that might go incredibly wrong"...

Re: So You Want to Write Your Own CSV code

#99

Post by daigoba66 »

huherto wrote: post #61

CSV works for simple cases. It is trivial to parse, you shouldn't even need a library. It there are many "what ifs" like in the posted article. You probably need another format like JSON (preferably) or XML.

Off topic, but why JSON over XML? What are the technical advantages for using JSON instead of XML (and don't say anything about "human readable"). If you're consuming the data with JavaScript, I'll grant you that JSON has quite an edge. But most every language has standard libs for XML. Both are easy to parse, but XML is easier to validate given a schema definition.

Re: So You Want to Write Your Own CSV code

#100

Post by bartonfink »

daigoba66 wrote: post #99
huherto wrote: post #61

CSV works for simple cases. It is trivial to parse, you shouldn't even need a library. It there are many "what ifs" like in the posted article. You probably need another format like JSON (preferably) or XML.

Off topic, but why JSON over XML? What are the technical advantages for using JSON instead of XML (and don't say anything about "human readable"). If you're consuming the data with JavaScript, I'll grant you that JSON has quite an edge. But most every language has standard libs for XML. Both are easy to parse, but XML is easier to validate given a schema definition.

Because of closing tags, XML is approximately twice as noisy as JSON encoding. If response sizes and the network traffic they entail are a concern, JSON is worth considering. If response sizes + validation are necessary, something like Protocol Buffers or Thrift may also fit, as they are widely supported as well.
Post reply on HN