Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Time to retire the CSV?

bitsondisk.com

Post reply on HN
Print view View original

491–500 of 594 posts

Re: Time to retire the CSV?

#491

Post by worik »

What a crock.

The author completely misses the point of what CSV files are useful for.

They are useful when both ends of the communication understand the context. They know what the data types are, they know what the character encoding is etcetera.

This is a very common situation. CSV files are easy to process, easy to generate, and can be read by a human without too much bother (they are not "for people" as the author so irritatingly asserts). I have written so many CSV (and other delimiters, not just ',') generators and processors I lost count decades ago. It is so easy.

And what does "retire CSV" really mean? Just stop generating them then! The consumers of your data will probably insist you go back to them as writing all that code just to satisfy your fetish with metadata is not useful

Now I am whinging.....

Re: Time to retire the CSV?

#492

Post by worik »

hermitcrab wrote: post #342
gsich wrote: post #158

Earlier quoted context omitted.

If you interpret "CSV" as purely comma seperated values then maybe. But in my bubble "CSV" means textfiles that are separated by some separator. Be it tabs, spaces, commas, or any other ASCII character. Some are more usable then others, if you have commas in your data then use tabs. If you have tabs use Form Feed or Record Separator or vertical tabs ... and so on. Of course this is not always applicable, since you so…

It seems quite common in some European countries to use semi-colons as the delimiter instead of commas (because they use commas as the decimal separator?), adding a new level of fun to parsing. In Easy Data Transform we count the number of commas, semi-colons, tabs in the file to make an educated guess as the delimiter (which the user can override).

Pipe '|' here

Re: Time to retire the CSV?

#493

Post by worik »

mortehu wrote: post #380
veltas wrote: post #369

Earlier quoted context omitted.

>CSV is actually horrible at parse-time vs. other data formats I find this really hard to believe given it's a simple enough syntax. And parsing is usually not the limiting factor, usually fast enough to not be noticed alongside interpreting or loading the source data. Every (much more sophisticated) compiler I can think of uses a linear parser based on this assumption.

Just extracting string views can be fast, but converting numbers to and from text is very slow.

It is a text format. So you will always have to do that conversion.

Binary formats have their own pain points - more of them actually

Re: Time to retire the CSV?

#494

Post by a1369209993 »

divbzero wrote: post #373
hermitcrab wrote: post #332

Earlier quoted context omitted.

>the fact that both of its separators (newlines and commas) can appear as-is inside column values, with a different meaning, if those column-values are quoted, means that there's no way to parallelize CSV processing, because there's no way to read-ahead and "chunk" a CSV purely lexically Yes, this is a major pain. It can be avoided by using Tab separated value (TSV) files, which don't use escaping. But then you can't…

ASCII has special delimiters 0x1E Record Separator and 0x1F Unit Separator to avoid conflicting with values, but they have never gained widespread adoption.

> but they have never gained widespread adoption.

Yes, because that just kicks the can down the road: how do you represent the one-byte string "\x1E"?

Re: Time to retire the CSV?

#495

Post by shock-value »

segmondy wrote: post #429
shock-value wrote: post #302

Earlier quoted context omitted.

Strong agree. TSV is better than CSV always (unless you need to interface with an external system that doesn't accept it). For those not aware, TSV and CSV differ by more than just the delimiting character. TSV has a dead-simple specification: https://www.iana.org/assignments/media-types/text/tab-separa... . CSV does not have a standard spec and implementations differ quite a bit, but often in subtle ways.

There's no difference between TSV and CSV but the separator, most good libraries allow you to use any character you wish as the separator. Here's the RFC for CSV - https://datatracker.ietf.org/doc/html/rfc4180

> There's no difference between TSV and CSV but the separator

This is not true, though almost everyone (understandably) assumes this based on the names.

Re: Time to retire the CSV?

#496

Post by Antrikshy »

worik wrote: post #491

What a crock. The author completely misses the point of what CSV files are useful for. They are useful when both ends of the communication understand the context. They know what the data types are, they know what the character encoding is etcetera. This is a very common situation. CSV files are easy to process, easy to generate, and can be read by a human without too much bother (they are not "for people" as the auth…

I stopped reading, so I may have missed it, but do they even address situations where humans interact with systems by supplying them with tabular data?

Re: Time to retire the CSV?

#497

Post by BenjiWiebe »

a1369209993 wrote: post #494
divbzero wrote: post #373

Earlier quoted context omitted.

ASCII has special delimiters 0x1E Record Separator and 0x1F Unit Separator to avoid conflicting with values, but they have never gained widespread adoption.

> but they have never gained widespread adoption. Yes, because that just kicks the can down the road: how do you represent the one-byte string "\x1E"?

At least very few humans type that into their spreadsheet cell.

Re: Time to retire the CSV?

#500

Post by ineedasername »

Under many scenarios it's not currently possible. As one example, I use SQL Developer extensively, and neither its XML or JSON exports include metadata, So that's no more useful than CSV. Excel is a no-go for anything with > 1M rows. Exporting large datasets leaves me with CSV since it's easier for recipients to parse than XML or JSON without additional work. And querying Oracle from within Python or R suffers a big performance hit over doing it directly from SQL developer, so for one-off projects I don't bother building a pipeline. (Repeat projects I'll build and schedule for downtime, or on a different PC.)
Post reply on HN