Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Time to retire the CSV?

bitsondisk.com

Post reply on HN
Print view View original

361–370 of 594 posts

Re: Time to retire the CSV?

#361

Post by the_only_law »

xtracto wrote: post #78
fighterpilot wrote: post #32

Earlier quoted context omitted.

I'm a human and I love CSVs. What other format can I open in a spreadsheet, access through cat/grep/awk, and easily load into any programming language? Any other format has to sacrifice one of these three things, and that's bad. It's this trifecta that makes it so versatile and human friendly.

> access through cat/grep/awk, and easily load into any programming language Until the CSV fields contain commas themselves. Even if fields are surrounded by "".

I recall one of my old Phones could dump SMS messages as CSV. I wanted to be build and interface that would let me search through my messages.

Turns out when you have data that contains both commas and quotes things get screwey real quick. You could have quoted data like:

123,ABC,”,””,456

Where ,” is column data.

I think that standard method is to double quote the field, but the dump sure wasn’t doing that for me, plus what happens when the data is something like abc””,?

Re: Time to retire the CSV?

#362

Post by rendall »

turtlebits wrote: post #318
asdff wrote: post #284

Earlier quoted context omitted.

Can you give an example? The rules for CSV files are so simple I'm struggling to imagine a case where something looks correct but in fact isn't correct.

Non standard delimiters. Escaping delimiters in fields - sometimes with a \, sometimes doubled (""), sometimes not at all. Double new lines. Poor handling from standard CSV libraries. Either unable to read or unable to create for some downstream process.

That sounds like the problem of badly formatted CSV, not a problem with CSV per se.

If you stick to one delimiter, and that delimiter is a comma, and escape the delimiter in the data with double-quotes around the entry, and escape double quotes with two double-quotes, well, you have written CSV that is correct and looks correct and will be parsed correctly by literally every CSV parser.

Re: Time to retire the CSV?

#364

Post by bhawks »

CSV is the embodiment of worse is better axiom.

Anyone that has dealt with more than 1 CSV is aware of many of the aspects of their horrible nature.

The more interesting reason is why they're so damn successful.

1: network effect - not supporting CSV in a product is practically silly. everyone can do it why can't you?

2: ease of producing / consuming (not saying that you do it correctly in all cases :))

3: data is transparent (or feels transparent)

4: accessible - if you can write language X you can parse or generate reasonable CSV in a mater of minutes. No knowledge of any libraries or tech required

That said it's horrible - but it will always be with us.

Re: Time to retire the CSV?

#365

Post by astine »

woodrowbarlow wrote: post #301
asdff wrote: post #284

Earlier quoted context omitted.

Can you give an example? The rules for CSV files are so simple I'm struggling to imagine a case where something looks correct but in fact isn't correct.

how're,you,handling,quotes?'

If you're manually generating your own CSV files, you probably know what kind of data you are generating and consequently whether your data is going to contain commas. If commas and newlines don't exist in your data, then you can safely ignore quoting rules when generating CSV files. I know that I've generated CSVs in the past and rather than figuring out the correct way to quote the strings, I just removed any inconvenient characters without any loss to the data at all. Obviously this is not "correct" but you don't have to implement cases if you know they won't show up.

Re: Time to retire the CSV?

#366

Post by 7thaccount »

Sunspark wrote: post #23

I don't agree with giving up csvs until the following conditions are met: 1) A truly open format is available and accessible. Csvs are textfiles. There is no system around that cannot open a textfile. If the format is binary or requires patents or whatever, then it's a non-starter. 2) Applications have a speed increase from using csvs. To wit, I loved csvs because often they finish preparing much faster than a "forma…

CSV is far from perfect, but it's nice that I can easily work with them without needing any libraries. All I need is file I/O and the ability to split strings. It doesn't get much simpler.

I'll admit though that "import JSON" and then being able to essentially convert the entire file into a dictionary is nice if the data has more structure to it.

Re: Time to retire the CSV?

#367

Post by munk-a »

postalrat wrote: post #292
unnouinceput wrote: post #273

Earlier quoted context omitted.

sqlite files are binary files, so a non-starter. If I open them in Notepad I can't read the values like I can with CSV.

"If I open them in Notepad" Well there's your problem.

There is a definite demand for some import format that you can trivially edit - CSV excels at this right now and JSON isn't that bad. Binary dump formats definitely do have a time and a place but there is also a separate need for trivially human readable formats.

Re: Time to retire the CSV?

#368

Post by m1234_ »

As someone who works for a company in the finance sector that provides services that several large banks, funds and financial institutions use - I can say with almost certainty that CSVs won't go away anytime soon. Most of our data sharing from and to these companies is via CSV files.

Re: Time to retire the CSV?

#369

Post by veltas »

derefr wrote: post #74
Sunspark wrote: post #23

I don't agree with giving up csvs until the following conditions are met: 1) A truly open format is available and accessible. Csvs are textfiles. There is no system around that cannot open a textfile. If the format is binary or requires patents or whatever, then it's a non-starter. 2) Applications have a speed increase from using csvs. To wit, I loved csvs because often they finish preparing much faster than a "forma…

You're comparing CSVs to other spreadsheet document formats. But a CSV is not a spreadsheet. A CSV is raw data. (It's data that is restricted to a shape that enables it to be easily imported into a spreadsheet—but data nevertheless.) As such, it should be compared to other data formats—e.g. YAML, JSON Lines, etc. These other data formats all win on your #2 against CSV, as CSV is actually horrible at parse-time vs. ot…

>CSV is actually horrible at parse-time vs. other data formats

I find this really hard to believe given it's a simple enough syntax. And parsing is usually not the limiting factor, usually fast enough to not be noticed alongside interpreting or loading the source data. Every (much more sophisticated) compiler I can think of uses a linear parser based on this assumption.

Re: Time to retire the CSV?

#370

Post by dtparr »

unnouinceput wrote: post #276
osigurdson wrote: post #99

Earlier quoted context omitted.

Take an Excel file and change the extension to .zip, then extract the contents. You will see that it is a collection of XML files. Therefore it should be reasonable to conclude that this approach can work for Excel sized datasets. However it is not particularly readable/diff-able if this is part of your use case.

Correction: the new xlsx is a zip file, the old xls format is true binary.

To be fair, xlsx came out with Office 2007, so it's not exactly 'new' anymore. Perhaps at this point it's reasonable for 'excel file' to mean the one that's been the default for 14 years?
Post reply on HN