Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Time to retire the CSV?

bitsondisk.com

Post reply on HN
Print view View original

121–130 of 594 posts

Re: Time to retire the CSV?

#121

Post by cogman10 »

fxleach wrote: post #88
cogman10 wrote: post #64

Earlier quoted context omitted.

There are probably valid uses for CSV, but more often than not, it's the wrong choice. As soon as data has any form of structure to it (and most data does). CSV complicates everything. Even for unstructured data, the problem of escape characters often shows it's ugly head. The moment your data contains a comma, tab, or space, you run into a nasty mess that, in the best case makes your system fail, and in the worst ca…

Just an FYI, JSON and XML both have characters that need to be escaped properly.

Correct, and those methods of escaping are clearly defined in both specs. In CSV, there is no universal escaping system. It's all over the board.

Any variable length text data format is going to run into issues when special characters are used in the data. CSV has no definition of what should happen when that occurs.

Re: Time to retire the CSV?

#122

Post by zabzonk »

masklinn wrote: post #97
zabzonk wrote: post #61

As the author of a CSV munging tool (CSVfix) I think most of the problems with CSV could be fixed if people producing CSV output, and people reading CSV input obeyed the rules of the RFC. Sadly, most people don't, and any textual output or input is routinely described as CSV, when it is nothing of the sort - even to the extent of not being comma-separated!

> Sadly, most people don't And the important think to remember is that you can not and will not make them.

No, but you could possibly suggest to them that what they are producing is not CSV. Otherwise, any sort of description becomes meaningless.

Re: Time to retire the CSV?

#123

Post by cbsmith »

Sunspark wrote: post #23

I don't agree with giving up csvs until the following conditions are met: 1) A truly open format is available and accessible. Csvs are textfiles. There is no system around that cannot open a textfile. If the format is binary or requires patents or whatever, then it's a non-starter. 2) Applications have a speed increase from using csvs. To wit, I loved csvs because often they finish preparing much faster than a "forma…

1. open(2) doesn't care about the file format

2. Yes, mixing formatting with data slows down data processing, don't do it.

3. Excel is not the replacement for CSV, and CSV is not a compact format. I mean, maybe if you are used to XML it is, but otherwise, just no.

Yes, there is truly a problem.

Re: Time to retire the CSV?

#124

Post by dragonwriter »

Sunspark wrote: post #23

I don't agree with giving up csvs until the following conditions are met: 1) A truly open format is available and accessible. Csvs are textfiles. There is no system around that cannot open a textfile. If the format is binary or requires patents or whatever, then it's a non-starter. 2) Applications have a speed increase from using csvs. To wit, I loved csvs because often they finish preparing much faster than a "forma…

> The author is not forced to use csvs.

The author appears to be a consultant selling data prep/transformation services. As long as the market is using CSVs, he’s forced to use CSVs, at least as end-of-pipeline inputs and outputs.

Of course, “people optimize their workflows for something other than making my job easy” is a common, but also rarely persuasive in motivating action from others with different jobs, complaint.

Re: Time to retire the CSV?

#125

Post by kaiju0 »

derefr wrote: post #74
Sunspark wrote: post #23

I don't agree with giving up csvs until the following conditions are met: 1) A truly open format is available and accessible. Csvs are textfiles. There is no system around that cannot open a textfile. If the format is binary or requires patents or whatever, then it's a non-starter. 2) Applications have a speed increase from using csvs. To wit, I loved csvs because often they finish preparing much faster than a "forma…

You're comparing CSVs to other spreadsheet document formats. But a CSV is not a spreadsheet. A CSV is raw data. (It's data that is restricted to a shape that enables it to be easily imported into a spreadsheet—but data nevertheless.) As such, it should be compared to other data formats—e.g. YAML, JSON Lines, etc. These other data formats all win on your #2 against CSV, as CSV is actually horrible at parse-time vs. ot…

Most data warehouses I have worked with use character delimited formats.

*shudder

Re: Time to retire the CSV?

#126

Post by javajosh »

zabzonk wrote: post #61

As the author of a CSV munging tool (CSVfix) I think most of the problems with CSV could be fixed if people producing CSV output, and people reading CSV input obeyed the rules of the RFC. Sadly, most people don't, and any textual output or input is routinely described as CSV, when it is nothing of the sort - even to the extent of not being comma-separated!

Is there something like CSVlint? Would be useful particular in the output stage, a kind of functional runtime test to make sure your program's output is valid and consumable.

Re: Time to retire the CSV?

#127

Post by _moof »

This is an example of a genre I like to describe as: programmer objects to a solution that meets everyone's requirements because it doesn't make them feel like a beautiful code-poet.

I like elegance as much as anyone. And I think it's a good proxy for other important qualities. But don't prioritize it above building something that actually does the job. Be an engineer.

Re: Time to retire the CSV?

#128

Post by dragontamer »

ARandomerDude wrote: post #108
dragontamer wrote: post #100

Earlier quoted context omitted.

> A truly open format is available and accessible. Sqlite? > Applications have a speed increase from using csvs. Sqlite? > The new format should not be grossly larger than the one it is replacing Sqlite it is. -------- Oh, you mean something that Excel can open? Oh yeah, I guess CSV then. But lets not pretend #1 (openness), #2 (speed), and #3 (size) are the issues.

I'd add the constraint that it needs to be understandable by most people. A typical non-tech office worker can use CSV without any trouble. Sqlite has an extremely steep learning curve for someone unfamiliar with databases.

Well yeah, that's kind of my point though.

We programmers get stuck into thinking that openness, speed, or other performance characteristics matter. But CSVs prove otherwise.

The reason why CSVs win in many situations is

1. Because they're convenient to edit.

2. Because they work in Microsoft Excel

CSVs have a myriad of issues: too many to list. But that doesn't matter, because convenience + Excel compatibility are king in practice.

To muse about other benefits is ignoring the core reason why CSV is used.

Re: Time to retire the CSV?

#129

Post by nonameiguess »

derefr wrote: post #74
Sunspark wrote: post #23

I don't agree with giving up csvs until the following conditions are met: 1) A truly open format is available and accessible. Csvs are textfiles. There is no system around that cannot open a textfile. If the format is binary or requires patents or whatever, then it's a non-starter. 2) Applications have a speed increase from using csvs. To wit, I loved csvs because often they finish preparing much faster than a "forma…

You're comparing CSVs to other spreadsheet document formats. But a CSV is not a spreadsheet. A CSV is raw data. (It's data that is restricted to a shape that enables it to be easily imported into a spreadsheet—but data nevertheless.) As such, it should be compared to other data formats—e.g. YAML, JSON Lines, etc. These other data formats all win on your #2 against CSV, as CSV is actually horrible at parse-time vs. ot…

It's not possible to read ahead and chunk with 100% assurance it will always work, but libraries like pandas and R's data.table do a reasonable job of reading in the first X rows, doing some usually correct type inference on the columns, and then read ahead and chunking the rest of the rows.

For what it's worth, I totally agree something like compressed json lines is a better data exchange format, but part of why csv remains as universal and supported as it is is that so much existing data storage applications export to either csv or excel and that's about it. So any ETL system that can't strictly control the source of its input data has no choice but to support csv.

Re: Time to retire the CSV?

#130

Post by nightski »

turtlebits wrote: post #109
radmuzom wrote: post #28

Every few years an article like this pops up. I find it tiring - because they are primarily from a software engineer's viewpoint who is probably trying to write a parser and needs to handle the edge cases. As a data scientist, I receive and process around 75GB of CSV every day - of course I don't process it manually. Our processes have been running a few years now and millions of dollars of revenue rides on it. I don…

It'll work until someone upstream upgrades their CSV writing library and your process breaks. But it's still writing CSV.

So you can constrain what type of CSV you will allow and if this happens it will bail. It's that simple. There is nothing wrong with having additional constraints on top of just saying it must be "CSV" especially in these scenarios.

I'm in a similar situation, we've been using CSV for over a decade to move billions of dollars worth of product each year. It just works.

Post reply on HN