Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Time to retire the CSV?

bitsondisk.com

Post reply on HN
Print view View original

541–550 of 594 posts

Re: Time to retire the CSV?

#541

Post by stickfigure »

lmm wrote: post #520
stickfigure wrote: post #311

Earlier quoted context omitted.

You misunderstood my point. CSV, for all its faults (and I am intimately aware of them), is ubiquitous. You're not going to replace a widespread standard with something that only offers a tiny incremental improvement. Ubiquitous is a feature, and generally trumps all other features. If you want something better than CSV to take off, at the very least it needs to solve the common pain points of CSV. JSON doesn't - all…

> If you want something better than CSV to take off, at the very least it needs to solve the common pain points of CSV. JSON doesn't - all it brings to the table is the ability to distinguish number vs text. That's a yawn. It brings the ability to parse in parallel - that's a big deal. And while number vs text might not be a huge difference in theory, in practice it eliminates what, 95% of real-world parse problems?

At least in my experience, I don't see a lot of trouble with number vs string in CSV data. You convert the string to a number or you don't. The pain points are usually dates or currencies - the same problem I usually have with JSON, because there's no standard format.

I think you could fix most of the pain of CSV simply by adding a second header row which defines the type of the column, using a common vocabulary of types. TEXT, NUMBER, ISO8861-DATE, etc.

Re: Time to retire the CSV?

#542

Post by Fwirt »

hermitcrab wrote: post #332
derefr wrote: post #74

Earlier quoted context omitted.

You're comparing CSVs to other spreadsheet document formats. But a CSV is not a spreadsheet. A CSV is raw data. (It's data that is restricted to a shape that enables it to be easily imported into a spreadsheet—but data nevertheless.) As such, it should be compared to other data formats—e.g. YAML, JSON Lines, etc. These other data formats all win on your #2 against CSV, as CSV is actually horrible at parse-time vs. ot…

>the fact that both of its separators (newlines and commas) can appear as-is inside column values, with a different meaning, if those column-values are quoted, means that there's no way to parallelize CSV processing, because there's no way to read-ahead and "chunk" a CSV purely lexically Yes, this is a major pain. It can be avoided by using Tab separated value (TSV) files, which don't use escaping. But then you can't…

The other advantage not being discussed here is that once you have a csv parser, it can be trivially generalized to use any character as a delimiter, and you can usually find a character that isn't found in the data you're working with unless you're working with binary data. I work on a legacy system where passing delimited files around is usually the integration method of the day, and if you can't delimit with commas (common) then you just delimit with tabs. If you can't delimit with tabs or commas, then delimit with pipes, which are rarely if ever used outside of code. Or use tildes, or grave accents, or...

Doing this avoids quoting, which avoids an entire class of problems. And yes, if you're passing multi-megabyte files around on a regular basis, or transferring giant blobs of text or binary data, or looking for a storage format with metadata then csvs are an absolutely awful choice, but as with any problem you pick the right tool for the job, and csvs are a great tool when you want to easily move some structured alphanumeric data around in a format that's trivial to parse in any language and widely supported.

Re: Time to retire the CSV?

#543

Post by vertere »

HanaShiratori wrote: post #476
postalrat wrote: post #406

Earlier quoted context omitted.

I guess my point is that CSVs or any text files aren't really "human readable". You still need some application top view them.

So what's the alternative? Good old pen and paper?

Doesn't count, you still need light to make it human-readable.

/s

Re: Time to retire the CSV?

#544

Post by danellis »

crazygringo wrote: post #529
danellis wrote: post #523

Earlier quoted context omitted.

What are all these hypothetical humans doing with CSV files? I've worked with them a lot, and the only time I read one manually is to find out why it isn't parsing properly. I'm not writing them by hand. I'm not extracting data from them by hand. What are other people doing?

Mainly Ctrl+F and diffs, in my case. Sometimes you need to look up a single code in a 50 MB file. And sometimes you need a quick check to see if one line or a million lines changes. It's "exception not the rule" type stuff... but it sure comes in handy to be able to check this stuff quick with basic text tools than have to run it through some binary parser. Same as JSON. But unlike protobufs for example.

Interesting. I would reach for `grep` and `diff` in those cases, so it's not much of a leap to imagine similarly using a tool for a binary format. After all, I only need the textual representation of the thing I'm searching for, not the entire dataset.

Re: Time to retire the CSV?

#545

Post by hermitcrab »

Fwirt wrote: post #542
hermitcrab wrote: post #332

Earlier quoted context omitted.

>the fact that both of its separators (newlines and commas) can appear as-is inside column values, with a different meaning, if those column-values are quoted, means that there's no way to parallelize CSV processing, because there's no way to read-ahead and "chunk" a CSV purely lexically Yes, this is a major pain. It can be avoided by using Tab separated value (TSV) files, which don't use escaping. But then you can't…

The other advantage not being discussed here is that once you have a csv parser, it can be trivially generalized to use any character as a delimiter, and you can usually find a character that isn't found in the data you're working with unless you're working with binary data. I work on a legacy system where passing delimited files around is usually the integration method of the day, and if you can't delimit with comma…

>it can be trivially generalized to use any character as a delimiter

But then you have the additional problem of trying to work out what the delimiter is. You can take an educated guess, but this isn't going to be 100% reliable.

Re: Time to retire the CSV?

#546

Post by red_admiral »

Sunspark wrote: post #23

I don't agree with giving up csvs until the following conditions are met: 1) A truly open format is available and accessible. Csvs are textfiles. There is no system around that cannot open a textfile. If the format is binary or requires patents or whatever, then it's a non-starter. 2) Applications have a speed increase from using csvs. To wit, I loved csvs because often they finish preparing much faster than a "forma…

For most of my use cases, I also need

4) can easily interact with Excel.

A lot of things 'around here' run off spreadsheets, because relational databases weren't invented here and no-one ever needs more than a few million rows apparently.

I seem to spend half my job on the current project exporting stuff to CSV, running it through my code, then opening the resulting CSV in excel and formatting it a bit and saving as .xlsx again.

Still, at least I don't have to use Visual Basic that way.

Re: Time to retire the CSV?

#547

Post by scbrg »

lanstin wrote: post #525
cbsmith wrote: post #498

Earlier quoted context omitted.

? Quite the contrary. It's more, "all the ones that any craftsman should be using".

A lot of excels and csvs are made by people other than programming crafters.

As with pretty much everything else computing, the world suffers because Microsoft has been dumping terrible tools on it for decades, and people just take their garbage as the way things have to be.

Re: Time to retire the CSV?

#548

Post by Ekaros »

kryptiskt wrote: post #147

I have always wondered why there is so little use of the separators in ASCII (the file, group, record and unit separators with codes 28-31). They seem perfect for the job and it would be easy to forbid inclusion of those characters in fields.

It's sad that so much ascii space is wasted on characters that are used... Really should replace that standard with something better...

Re: Time to retire the CSV?

#549

Post by vertere »

stickfigure wrote: post #541
lmm wrote: post #520

Earlier quoted context omitted.

> If you want something better than CSV to take off, at the very least it needs to solve the common pain points of CSV. JSON doesn't - all it brings to the table is the ability to distinguish number vs text. That's a yawn. It brings the ability to parse in parallel - that's a big deal. And while number vs text might not be a huge difference in theory, in practice it eliminates what, 95% of real-world parse problems?

At least in my experience, I don't see a lot of trouble with number vs string in CSV data. You convert the string to a number or you don't. The pain points are usually dates or currencies - the same problem I usually have with JSON, because there's no standard format. I think you could fix most of the pain of CSV simply by adding a second header row which defines the type of the column, using a common vocabulary of t…

I envy your life of pristine CSVs and compatible implementations, where JSON doesn't solve any problems. But CSVs have a broad range of uses, and there are, in fact, plenty of implementations that do not follow RFC 4180, and even if everything did there would still be issues like the header row being optional (without an in-band way of telling). JSON is, in practice, much more well-defined.

On text vs numbers, at least some widely-used software (e.g. R, Excel) will try to guess for you. It should be obvious how this might cause problems. Maybe one should turn auto-conversion off (or not use things that don't let you turn it off) and specify which columns are numbers. Some datasets have a lot of columns, so this can be a PITA, even if you do know which ones should be numbers. But the bigger problem is if you have to deal with other people, or the files that they've touched. There are always going to be people that edit data in excel, don't use the right options when they import, etc.

Re: Time to retire the CSV?

#550

Post by rendall »

hnick wrote: post #515
rendall wrote: post #506

Earlier quoted context omitted.

> It is very easy to overlook edge cases in CSV. That was not a criticism from the original article, and isn't even true. If you have newlines in your original data, and you "overlooked" this "edge case", then neither JSON nor YAML nor any other format will save you. The same fix applies to them all. This is really a very poor criticism.

The only reason I replied was that you said: > The algorithm for creating well-formed CSV from data is straightforward and almost trivial: if the datum has no comma in it, leave it alone. It's good to go. > Not complicated and covers every edge case Maybe more context was implied, but I didn't want anyone to think it's that simple. I have received and had to process CSV data with unexpected new lines. It gets nearly…

That is a fair point :)
Post reply on HN