Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Time to retire the CSV?

bitsondisk.com

Post reply on HN
Print view View original

401–410 of 594 posts

Re: Time to retire the CSV?

#401

Post by flowerlad »

CSV isn't going away any time soon. It is the only format that is supported by all data-oriented tools. Getting all those tools to agree on an alternative isn't going to happen, especially given that that there are all these old tools that are no longer being updated, that produce and consume CSV.

Instead of finding a replacement for CSV it might be easier to standardize it and enhance it. Excel's version of CSV is the de-facto standard. If you want a written-down spec that is available too [1]. To this we need to add enhancements such as a way to specify metadata (i.e., data type of each field). No need to find an alternative to CSV!

[1] https://datatracker.ietf.org/doc/html/rfc4180

Re: Time to retire the CSV?

#402

Post by cbsmith »

hermitcrab wrote: post #338
wombatpm wrote: post #288

Earlier quoted context omitted.

Of course there is a old solution in the ANSI character set. File, Record, Group and Unit separator characters

Yes. You could get a long way with a text format in which: -the first line is always a header -fields are separated by Unit separator characters -records are separated by Record separator characters -encoding is UTF8 If you wanted to get fancy you could also have: -comment lines -column metadata (e.g. column 0 is an ISO date, column 2 is text, column 3 is an integer) Both the above could start with a Unicode characte…

> it wouldn't be easy to create/edit manually

I mean, you'd have to be using a pretty terrible tool for it not being able to handle that, and I suspect if such an approach were to become prevalent, that tool would either fix the glitch or become largely unused.

Re: Time to retire the CSV?

#403

Post by anigbrowl »

osigurdson wrote: post #99
derefr wrote: post #74

Earlier quoted context omitted.

You're comparing CSVs to other spreadsheet document formats. But a CSV is not a spreadsheet. A CSV is raw data. (It's data that is restricted to a shape that enables it to be easily imported into a spreadsheet—but data nevertheless.) As such, it should be compared to other data formats—e.g. YAML, JSON Lines, etc. These other data formats all win on your #2 against CSV, as CSV is actually horrible at parse-time vs. ot…

Take an Excel file and change the extension to .zip, then extract the contents. You will see that it is a collection of XML files. Therefore it should be reasonable to conclude that this approach can work for Excel sized datasets. However it is not particularly readable/diff-able if this is part of your use case.

That's useful to know.

Re: Time to retire the CSV?

#404

Post by AussieWog93 »

pwinnski wrote: post #215

This reads like a joke. If you think you can do better than CSV, let's see your proposal. Hint: you probably can't, and if you could, you probably couldn't get Excel to export it, so you still probably can't. "The status quo is bad, more recent popular formats aren't good enough either, but I don't actually have a specific proposal that's better than all of the above" is a lot faster to read than that article, and sa…

>If you think you can do better than CSV, let's see your proposal.

I've got one! It's basically the same as regular CSV, but everything is UTF-8, the columns and lines are delineated by dedicated UTF-8 "delineator" codepoints (if they aren't defined in the spec, find reasonable surrogates and use them), and therefore nothing ever needs to be escaped.

More human readable than regular CSV, less prone to error and just as easy to write to in a for loop (easier, in fact, as there are no escapes).

Depending on how Excel handles delineators, it should be able to import it too.

Re: Time to retire the CSV?

#405

Post by sicariusnoctis »

wernercd wrote: post #118
Twisell wrote: post #53

Earlier quoted context omitted.

OP and you gave me an idea : "The only true successor of CSV should be forward/backward compatible with any existing CSV variant" If we manage to write a spec that meet this criteria we'll have a powerful standard with easy adoption.

obligatory: https://xkcd.com/927/

This is true. The best way would be if everyone generated CSV files that were compatible with on one or two already existing CSV formats. (e.g. no-quote-plain-old-csv or quotes-forced-csv.) Hopefully, it would be easy to distinguish between the two in some well-defined manner. Maybe call it .csvs (CSV standardized). Or just introduce new extensions like .csvq ("CSV quoted").

Note the difference is that I am suggesting reducing the number of standards in-use by using only one already existing CSV format. :)

Re: Time to retire the CSV?

#406

Post by postalrat »

munk-a wrote: post #367
postalrat wrote: post #292

Earlier quoted context omitted.

"If I open them in Notepad" Well there's your problem.

There is a definite demand for some import format that you can trivially edit - CSV excels at this right now and JSON isn't that bad. Binary dump formats definitely do have a time and a place but there is also a separate need for trivially human readable formats.

I guess my point is that CSVs or any text files aren't really "human readable". You still need some application top view them.

Re: Time to retire the CSV?

#407

Post by hermitcrab »

cbsmith wrote: post #402
hermitcrab wrote: post #338

Earlier quoted context omitted.

Yes. You could get a long way with a text format in which: -the first line is always a header -fields are separated by Unit separator characters -records are separated by Record separator characters -encoding is UTF8 If you wanted to get fancy you could also have: -comment lines -column metadata (e.g. column 0 is an ISO date, column 2 is text, column 3 is an integer) Both the above could start with a Unicode characte…

> it wouldn't be easy to create/edit manually I mean, you'd have to be using a pretty terrible tool for it not being able to handle that, and I suspect if such an approach were to become prevalent, that tool would either fix the glitch or become largely unused.

Are there any editors that let you insert a Unit separator character as easily as a comma?

Re: Time to retire the CSV?

#408

Post by hermitcrab »

mavhc wrote: post #398
hermitcrab wrote: post #389

Earlier quoted context omitted.

While these have clear advantages over comma and CR for parsing (no more escaping!), they aren't at all convenient for manual editing.

If they were popular they'd be supported in your editor though.

Maybe. It is a bit of a chicken-and-egg problem though. They aren't likely to become popular until editors support them.

Also, the fact that they have been around for years and never taken off implies that they probably never will.

Re: Time to retire the CSV?

#409

Post by gsich »

asdff wrote: post #237
gsich wrote: post #158

Earlier quoted context omitted.

If you interpret "CSV" as purely comma seperated values then maybe. But in my bubble "CSV" means textfiles that are separated by some separator. Be it tabs, spaces, commas, or any other ASCII character. Some are more usable then others, if you have commas in your data then use tabs. If you have tabs use Form Feed or Record Separator or vertical tabs ... and so on. Of course this is not always applicable, since you so…

Those other things have different names like TSV

Technically true, but like I said "CSV" is more a term for human readable data, with some delimiter in them. Maybe it's a comma, maybe not. In every case you need someonne to look at it. If you want a machine-machine data protocol you can use XML or JSON if it needs to be somewhat human readable.

Re: Time to retire the CSV?

#410

Post by anigbrowl »

karteum wrote: post #264
Sunspark wrote: post #23

I don't agree with giving up csvs until the following conditions are met: 1) A truly open format is available and accessible. Csvs are textfiles. There is no system around that cannot open a textfile. If the format is binary or requires patents or whatever, then it's a non-starter. 2) Applications have a speed increase from using csvs. To wit, I loved csvs because often they finish preparing much faster than a "forma…

I personally believe that at least SQLite matches all those criteria : "1) A truly open format is available" : sqlite is open-source, MIT-licensed, and well specified (even though I am usually not so happy with its weak typing approach, yet in this case this precisely enables a 100% correspondance between CSV and sqlite since CSV has also no typing at all...) "2) Applications have a speed increase from using csvs" :…

It's worth mentioning that SQLite is committed by its publishers to be backward-compatible out to 2050 and is the recommended by the Library of Congress as a safe long-term storage format (as are XML, JSON, and CSV). https://www.sqlite.org/locrsf.html

For single tables a database is probably overkill, but it's nice to have around when you need something reasonably powerful without being overly complex or hard to get started with.

Post reply on HN