Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

So You Want to Write Your Own CSV code

tburette.github.io

Post reply on HN
Print view View original

111–120 of 127 posts

Re: So You Want to Write Your Own CSV code

#111

Post by iagooar »

chrismcb wrote: post #72
iagooar wrote: post #60

"CSV is not a well defined file-format. The RFC4180 does not represent reality. It seems as every program handles CSV in subtly different ways. Please do not inflict another one onto this world. Use a solid library." I can't but disagree when I read stuff like this. Why shouldn't I release a library if I think it's good enough for the community? Even the powerful and versatile Ruby library for CSV parsing started as…

IF your library is a solid library, then release it. What he is saying though, is don't roll your own if you can use a solid library. And if a good solid library exists, why bother writing your own?

> And if a good solid library exists, why bother writing your own?

Because, you know, learning, having fun and stuff.

Re: So You Want to Write Your Own CSV code

#112

Post by zAy0LfpBZLC8mAC »

lignuist wrote: post #108
zAy0LfpBZLC8mAC wrote: post #105

Earlier quoted context omitted.

No, it doesn't. What if there is #COMMA# in one of the fields?

You just choose a placeholder that does not appear in the data. You could even implement it in a way that a placeholder is automatically selected upfront that does not appear in the data. When it comes to parsing, the thing is that you usually have to make some assumptions about the document structure.

What if there is #COMMA, in one of the fields (but no #COMMA#)?

Yes, the assumption you have to make is called the grammar, and you better have a parser that always does what the grammar says, and global text replacement is a technique that is easy to get wrong, difficult to prove correct, and completely unnecessary at that.

Re: So You Want to Write Your Own CSV code

#113

Post by jimeh »

Personally I know the pain of creating a CSV parser. In late 2006 I was working on a PHP project that required a CSV parser, and what was available at the time did not come close to cutting it. So I created my own parser/generator, which among many other things included automatic delimiter character detection. It was a rather painful project to create, but I learned a lot, and found the experience really fun.

Overall I agree with the article, there's no point in reinventing the wheel if there are libraries out there. And CSV specifically is a horribly complex format to deal with. But sometimes rolling your own is the best and/or only choice you have, and you might come out the other end enjoying the experience, and having learned a lot.

As for what happened to my old CSV parser? It ended up being quite popular, but stuck in the dark ages as I'd mostly moved on from PHP years ago. But thanks to a contributor, we've recently put renewed effort into bringing the project in to modern times: https://github.com/parsecsv/parsecsv-for-php

Re: So You Want to Write Your Own CSV code

#114

Post by lignuist »

zAy0LfpBZLC8mAC wrote: post #112
lignuist wrote: post #108

Earlier quoted context omitted.

You just choose a placeholder that does not appear in the data. You could even implement it in a way that a placeholder is automatically selected upfront that does not appear in the data. When it comes to parsing, the thing is that you usually have to make some assumptions about the document structure.

What if there is #COMMA, in one of the fields (but no #COMMA#)? Yes, the assumption you have to make is called the grammar, and you better have a parser that always does what the grammar says, and global text replacement is a technique that is easy to get wrong, difficult to prove correct, and completely unnecessary at that.

> What if there is #COMMA, in one of the fields (but no #COMMA#)?

What should happen? Since #COMMA is not #COMMA#, it gets not replaced, because it does not match.

Please keep in mind, that I replied to suni's very specific question and did not try to start a discussion about general parser theory. In practice, we find a lot of files that do not respect the grammar, but still need to find a way to make the data accessible.

Re: So You Want to Write Your Own CSV code

#115

Post by zAy0LfpBZLC8mAC »

lignuist wrote: post #114
zAy0LfpBZLC8mAC wrote: post #112

Earlier quoted context omitted.

What if there is #COMMA, in one of the fields (but no #COMMA#)? Yes, the assumption you have to make is called the grammar, and you better have a parser that always does what the grammar says, and global text replacement is a technique that is easy to get wrong, difficult to prove correct, and completely unnecessary at that.

> What if there is #COMMA, in one of the fields (but no #COMMA#)? What should happen? Since #COMMA is not #COMMA#, it gets not replaced, because it does not match. Please keep in mind, that I replied to suni's very specific question and did not try to start a discussion about general parser theory. In practice, we find a lot of files that do not respect the grammar, but still need to find a way to make the data acces…

What would happen is that you first would replace #COMMA, with #COMMA#COMMA# and then later replace that with ,COMMA# , thus garbling the data.

The way to make the data accessible is to request the producer to be fixed, it's that simple. If that is completely impossible, you'll have to figure out the grammar of the data that you actually have and build a parser for that. Your suggested strategy does not work.

Re: So You Want to Write Your Own CSV code

#116

Post by lignuist »

zAy0LfpBZLC8mAC wrote: post #115
lignuist wrote: post #114

Earlier quoted context omitted.

> What if there is #COMMA, in one of the fields (but no #COMMA#)? What should happen? Since #COMMA is not #COMMA#, it gets not replaced, because it does not match. Please keep in mind, that I replied to suni's very specific question and did not try to start a discussion about general parser theory. In practice, we find a lot of files that do not respect the grammar, but still need to find a way to make the data acces…

What would happen is that you first would replace #COMMA, with #COMMA#COMMA# and then later replace that with ,COMMA# , thus garbling the data. The way to make the data accessible is to request the producer to be fixed, it's that simple. If that is completely impossible, you'll have to figure out the grammar of the data that you actually have and build a parser for that. Your suggested strategy does not work.

I used that strategy for parsing gigabytes of CSVs containing arbitrary natural language from the web - try to get these files fixed, or figure out a grammar for gigabytes of fuzzy data...

My approach never failed for me, so telling me that my strategy does not work is a strong claim, where it reliably did the job for me.

Your examples are all valid, but what you are describing are theoretical attacks on the method, while the method works in almost all cases in practice. We are talking about two different viewpoints: dealing with large amounts of messy data on one hand and parser theory in an ideal cosmos on the other hand.

Re: So You Want to Write Your Own CSV code

#118

Post by Mister_Snuggles »

barrkel wrote: post #55
slg wrote: post #36

CSV are a headache. Like the article says, RFC4180 doesn't necessarily represent the real world. However sometimes you just have to reject things that aren't spec. Not too long ago I was struggling with one of these CSV issues and received some good advice from Hans Passant [1] on a Stack Overflow question pertaining to my problem (emphasis mine): "It is pretty important that you don't try to fix it. That will make y…

On the other hand, the ability to handle all kinds of input can be a chief selling point of your product. In my current job, the most common "invalid" CSV format we get is .xlsx files. So I wrote an .xlsx parser (way, way faster than Apache POI). Another interesting hiccup to consider is CSV inside individual fields - i.e. recursive CSV. There are various ways to handle this, but in my company's line of business the…

> Likely the next invalid format we'll have to parse is PDFs containing tables...

And after that you will have to parse PDFs containing scans (as images, not text) of pages containing tables...

Re: So You Want to Write Your Own CSV code

#119

Post by zAy0LfpBZLC8mAC »

lignuist wrote: post #116
zAy0LfpBZLC8mAC wrote: post #115

Earlier quoted context omitted.

What would happen is that you first would replace #COMMA, with #COMMA#COMMA# and then later replace that with ,COMMA# , thus garbling the data. The way to make the data accessible is to request the producer to be fixed, it's that simple. If that is completely impossible, you'll have to figure out the grammar of the data that you actually have and build a parser for that. Your suggested strategy does not work.

I used that strategy for parsing gigabytes of CSVs containing arbitrary natural language from the web - try to get these files fixed, or figure out a grammar for gigabytes of fuzzy data... My approach never failed for me, so telling me that my strategy does not work is a strong claim, where it reliably did the job for me. Your examples are all valid, but what you are describing are theoretical attacks on the method,…

How do you know that the strategy worked reliably if you never compared the results to the results obtained using a reliable method (which you presumably didn't, because then you could just have used the reliable method)? The larger the data you have to deal with, the more likely it is that corner cases will occur in it, and the less likely that you will notice anomalies, thus the more important that you are very strict in your logic if you want to derive any meaningful results.

As such, the two viewpoints really are: not really caring about the soundness of your results and solving the actual problem.

Now, maybe you really can show that the bugs in the methods you use only cause negligible noise in your results, in which case it might be perfectly fine to use those methods. But just ignoring errors in your deduction process because you don't feel like doing the work of actually solving the problem at hand is not pragmatism. You'll have to at least demonstrate that your approach does not invalidate the result.

Re: So You Want to Write Your Own CSV code

#120

Post by justifier »

i recently needed to deal with a ~4G xml file.. i tried a parser but after waiting thirty minutes for it to load i decided to parse out the bits i needed manually with a bash script

knowing my needs i could easily account for all possible muck ups and avoid the instances where ambiguity could play a part

i was then able to use the bits i pulled out of the ~4G file, now 16M, in the parser with all of its assurances

sure, edge cases justify using a tried and true library for generics, but there are also edge cases that justify mocking up your own naive implementation.. if only, like in my case, to make the dada usable in such a library

Post reply on HN