Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

So You Want to Write Your Own CSV code

tburette.github.io

Post reply on HN
Print view View original

21–30 of 127 posts

Re: So You Want to Write Your Own CSV code

#21

Post by keeperofdakeys »

mantrax5 wrote: post #6

Why are people using CSV when better (and less fuzzily defined) solutions exist, such as JSON?

If you have simple data, why use something as complicated as JSON? For a recent project, I had a simple CSV file with an int and float per row; using JSON would probably double the datasize. I used a simple string.split(",") for the javascript decoder, because I control the data, and know it's safe. I don't need another javascript library (I'd probably do differently if I had a standard library, not a hodge-podge of scripts).

Sometimes, simplicity is better for everyone.

Re: So You Want to Write Your Own CSV code

#22

Post by mrweasel »

The most retard structure I've seen in a CSV file relates to the "What if the character separating fields is not a comma?".

We get "CSV" files from Klarna, an invoicing company, with the payments they've processed for us. Because we're Danish and they are Swedish, it's not really weird that they would use comma as the decimal separator. So to compensate for having used the comma, they for some reason picks ", " ( that's comma + space ) as the field separator. Most good csv parsers can handle the field separator to be any character you like, as long is it's just ONE character. By picking a two character separator they've just dictated that I write my own or resort to just splitting a line on ", ".

Re: So You Want to Write Your Own CSV code

#23

Post by mooreds »

This goes for most complex problems. The first step of any dev problem should be to make sure you understand the problem, the second to map out the main pieces and the third to make sure you are leveraging every (well maintained) library possible. There are, of course, issues with dependencies and tying yourself to code you didn't write, but what would you rather depend on--code that has had tens or hundreds of eyes on it, or code that you, and maybe one or two team members has reviewed?

Re: So You Want to Write Your Own CSV code

#24

Post by mrweasel »

mantrax5 wrote: post #10
jiggy2011 wrote: post #9

Earlier quoted context omitted.

Importing into excel is probably a big reason.

If Excel compatibility is the goal, one should use libraries that read and produce Excel files. CSV is bullshit, it's not good for anything except scenarios where you control both the export process, and the parser (so you know what delimiter is used and so on).

CSV files are MUCH easier to search and inspect using tools like grep and less. It's the accounting people that want's Excel, but as a developer CSV is easier and more flexible.

We process csv files containing payment information, export csv as product feeds for perhaps 10 different partners. It works, it's faster than XML or JSON and is easy for non-technical people to inspect using tools like Excel. Most cvs data exports are not something that's made up on the fly every day, it batch jobs that create the same type of CSV file each and every time.

CSV is alright. XML is bullshit.

Re: So You Want to Write Your Own CSV code

#25

Post by foxhill »

as the article mentions, CSV is not well defined. libraries are.. well, different. you'd spend as much time becoming familiar with one as you would writing a basic parser.

commas don't delimit field entries? CSV -> comma separated values.

new lines inside a field? i've never written a parser that would be foiled by this. could be an issue if you use a built-in tokeniser (e.g strtok, etc.). be aware.

variable number of fields? you’re probably writing this for something with an expected input form. throw errors if you see something you do not accept. make sure you catch them.

ascii/unicode? yea. it’s a fucking mess. everywhere.

just do it. handle failure gracefully. learn from your mistakes. don't be naive. consider a library if the (risk of failure):(time) ratio is skewed the wrong way. the only time i would absolutely insist that a 3rd party library be used is when crypto is involved. even then, be aware that they are not perfect.

absolutely ignore people who's argument is along the lines of "you are not smart enough to implement this standard. let someone else do it.”. fuck everything about that statement, and it’s false sense of superiority.

nothing comes for free. wether you use a library, or do your own thing, you’re going to run into problems.

Re: So You Want to Write Your Own CSV code

#27

Post by tikumo »

mrweasel wrote: post #22

The most retard structure I've seen in a CSV file relates to the "What if the character separating fields is not a comma?". We get "CSV" files from Klarna, an invoicing company, with the payments they've processed for us. Because we're Danish and they are Swedish, it's not really weird that they would use comma as the decimal separator. So to compensate for having used the comma, they for some reason picks ", " ( tha…

it can be irritating, but you can just as easy parse ", " to "|" or something, by simple string replacing, pre parsing..

Re: So You Want to Write Your Own CSV code

#28

Post by joshvm »

I trust Numpy a lot for CSV handling. It deals with lots of edge cases including missing data, weird delimiters (pipes '|' are popular in astro for some reason) and massive files. If in doubt, whack it into Excel which has been doing this stuff for decades now. I prefer using Numpy to Python's CSV library which I find a bit clunky.

Very little data is actually true CSV.

The code isn't particularly long (~900 lines), it's Python (hence readable) and it's well commented:

https://github.com/numpy/numpy/blob/v1.8.1/numpy/lib/npyio.p...

Re: So You Want to Write Your Own CSV code

#29

Post by jasode »

joshvm wrote: post #28

I trust Numpy a lot for CSV handling. It deals with lots of edge cases including missing data, weird delimiters (pipes '|' are popular in astro for some reason) and massive files. If in doubt, whack it into Excel which has been doing this stuff for decades now. I prefer using Numpy to Python's CSV library which I find a bit clunky. Very little data is actually true CSV. The code isn't particularly long (~900 lines),…

>weird delimiters (pipes '|' are popular in astro for some reason)

I can only guess that since it's astronomy data and constellation coordinates have decimal places, it's best to avoid the comma character because some countries use it as a decimal separator.

http://en.wikipedia.org/wiki/Decimal_mark

Re: So You Want to Write Your Own CSV code

#30

Post by sunir »

tikumo wrote: post #27
mrweasel wrote: post #22

The most retard structure I've seen in a CSV file relates to the "What if the character separating fields is not a comma?". We get "CSV" files from Klarna, an invoicing company, with the payments they've processed for us. Because we're Danish and they are Swedish, it's not really weird that they would use comma as the decimal separator. So to compensate for having used the comma, they for some reason picks ", " ( tha…

it can be irritating, but you can just as easy parse ", " to "|" or something, by simple string replacing, pre parsing..

Think it through. What if there is free text in the field? "How are you, Sally?"
Post reply on HN