Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

So You Want to Write Your Own CSV code

tburette.github.io

Post reply on HN
Print view View original

101–110 of 127 posts

Re: So You Want to Write Your Own CSV code

#101

Post by lignuist »

Someone wrote: post #42
lignuist wrote: post #34

Earlier quoted context omitted.

You can replace all commas with a placeholder (e.g. "#COMMA#"), replace the delimiter with a comma, parse the document and then replace all placeholders in the data with ",".

That does not work, unless that first replacement magically ignores the commas that are part of field separators. If you know how to write the code that does that, your problem is solved.

I was referencing to "What if the character separating fields is not a comma?".

And there it clearly works. I used this technique a few times with success. If you find a CSV file that has mixed field separator types, then you probably found a broken CSV file.

Re: So You Want to Write Your Own CSV code

#102

Post by brianpgordon »

EpicEng wrote: post #59

> What if the character separating fields is not a comma? > Not kidding. We'd ll be better off really, but that ship has sailed. Using CSV for data which is only ever read by a machine is a dumb decision. Use the RS (record separator) character and many of these ambiguities disappear. Of course, like I said, that ship has sailed. If you want your data to be read nicely by other programs you're probably stuck with CSV…

On the other hand, there's definitely some value to being able to directly inspect and alter your data in a text editor. It would be nice to not have to deal with unprintable characters.

Re: So You Want to Write Your Own CSV code

#103

Post by [deleted] »

slg wrote: post #36

CSV are a headache. Like the article says, RFC4180 doesn't necessarily represent the real world. However sometimes you just have to reject things that aren't spec. Not too long ago I was struggling with one of these CSV issues and received some good advice from Hans Passant [1] on a Stack Overflow question pertaining to my problem (emphasis mine): "It is pretty important that you don't try to fix it. That will make y…

[deleted]

Re: So You Want to Write Your Own CSV code

#104

Post by Sami_Lehtinen »

Parsing CSV is easier than handling XML or JSON. I do integrations as my job and most common format used is CSV because it's handy simple and reliable compared to other formats. That is exactly the reason why ini and props file are also preferred over database for data which isn't too volatile or big. Any one can open the datafile and see what's stored and what's wrong.

Re: So You Want to Write Your Own CSV code

#105

Post by zAy0LfpBZLC8mAC »

lignuist wrote: post #101
Someone wrote: post #42

Earlier quoted context omitted.

That does not work, unless that first replacement magically ignores the commas that are part of field separators. If you know how to write the code that does that, your problem is solved.

I was referencing to "What if the character separating fields is not a comma?". And there it clearly works. I used this technique a few times with success. If you find a CSV file that has mixed field separator types, then you probably found a broken CSV file.

No, it doesn't. What if there is #COMMA# in one of the fields?

Re: So You Want to Write Your Own CSV code

#106

Post by userbinator »

muteh wrote: post #58
slg wrote: post #36

CSV are a headache. Like the article says, RFC4180 doesn't necessarily represent the real world. However sometimes you just have to reject things that aren't spec. Not too long ago I was struggling with one of these CSV issues and received some good advice from Hans Passant [1] on a Stack Overflow question pertaining to my problem (emphasis mine): "It is pretty important that you don't try to fix it. That will make y…

I've never really got my head round RFCs, but 4180 is only informational, not a standard. I have used exactly your argument before though, and will again. Have also been on the other side and needed to convert horribly inconsistent data to fit it.

It's not a standard but it should very well be one, in my opinion. Whenever someone has to process CSV I always point to it to make them aware of it, that a lot of subtle points like escaping have already been defined to be done in one way. There is no good reason to NOT follow RFC4180 if you want to produce/consume CSV.

Re: So You Want to Write Your Own CSV code

#107

Post by dceddia »

encoderer wrote: post #92

Early on in my career, just a year out of school, I, for some absurd reason, had the idea to build my own date library. Primarily, I didn't fully understand the date objects and functions available in the languages/libraries i was using so simple things like formatting a string date seemed difficult to me. This was an awful idea. Dreadful. I came up with all sort of delightful helper methods to cover common use cases…

Especially dates.

Re: So You Want to Write Your Own CSV code

#108

Post by lignuist »

zAy0LfpBZLC8mAC wrote: post #105
lignuist wrote: post #101

Earlier quoted context omitted.

I was referencing to "What if the character separating fields is not a comma?". And there it clearly works. I used this technique a few times with success. If you find a CSV file that has mixed field separator types, then you probably found a broken CSV file.

No, it doesn't. What if there is #COMMA# in one of the fields?

You just choose a placeholder that does not appear in the data. You could even implement it in a way that a placeholder is automatically selected upfront that does not appear in the data.

When it comes to parsing, the thing is that you usually have to make some assumptions about the document structure.

Re: So You Want to Write Your Own CSV code

#109

Post by qwerty_asdf »

Garbage in? Garbage out. You give me a shitty file, you get shitty results. Tough shit.

None of these questions are particularly daunting. CSV means "comma separated values", so if you want to play games and use other delimiters, please fuck off. If it's not a comma, then guess what: it's not delimited. New line characters are well-known, and well-understood, across all platforms and easy to detect. If you manage to fuck that up in your file, then take a look in the mirror, because the problem is you. Enforcing the practice of enclosing the target data in quotation marks among users is a good idea. It's something that should be supported and encouraged, and ignored at one's own risk.

Additionally, employing an escape character (such as backslash) to allow for the use of a quotation mark within enclosing quotation marks is a nice feature to add in. After that, the concept of a CSV file has provided enough tools, to tolerate [an arbitrarily large percentage] of all use cases. If you need something more robust, XML is thataway.

Re: So You Want to Write Your Own CSV code

#110

Post by skybrian »

daigoba66 wrote: post #99
huherto wrote: post #61

CSV works for simple cases. It is trivial to parse, you shouldn't even need a library. It there are many "what ifs" like in the posted article. You probably need another format like JSON (preferably) or XML.

Off topic, but why JSON over XML? What are the technical advantages for using JSON instead of XML (and don't say anything about "human readable"). If you're consuming the data with JavaScript, I'll grant you that JSON has quite an edge. But most every language has standard libs for XML. Both are easy to parse, but XML is easier to validate given a schema definition.

JSON is considerably more compact, especially if you use lists instead of maps. For a list of numbers, there is only one character of overhead per item. For a list of strings, it's three characters per item.

Of course you can embed comma-separated lists in XML, but with JSON it will parse them for you.

(And of course it's not as good as a protobuf, but not bad for a text format.)

Post reply on HN