Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Time to retire the CSV?

bitsondisk.com

Post reply on HN
Print view View original

531–540 of 594 posts

Re: Time to retire the CSV?

#531

Post by creata »

postalrat wrote: post #406
munk-a wrote: post #367

Earlier quoted context omitted.

There is a definite demand for some import format that you can trivially edit - CSV excels at this right now and JSON isn't that bad. Binary dump formats definitely do have a time and a place but there is also a separate need for trivially human readable formats.

I guess my point is that CSVs or any text files aren't really "human readable". You still need some application top view them.

In some incredibly pedantic sense, sure. But the difference is that every computer comes with what a human needs to read a CSV file. That's not true of an SQLite database.

Re: Time to retire the CSV?

#532

Post by newbie2020 »

fuhrysteve wrote: post #71
yuy910616 wrote: post #14

CSV is bad - but convenience is hard to beat. It's the same thing with excel. People abuse it, but you just can't beat the fact that your programmer, CEO, analyst, and secretary can all contribute to the same file.

Yeah, this is pretty much it. The author complains about CSVs being "notoriously inconsistent" as though switching to some other format would magically change that. They're only inconsistent because sometimes lazy programmers do ",".join(mylist) instead of using an RFC4180 compliant CSV writer. Lazy programmers will just use non-compliant methods of creating whatever magic format OP is dreaming about. Case in point:…

Hah, I use the join function when I write them. Can you elaborate upon why that’s bad?

Re: Time to retire the CSV?

#533

Post by xkgt »

kevin_thibedeau wrote: post #471
sli wrote: post #459

Earlier quoted context omitted.

More generally than editor support, how is the average person expected to type them? I could easily add them to my keyboard, but if "editor support" turned out to mean "Edit > Insert > Record Separator Character" and I was now bound only to editors that support this, absolutely nobody is going to bother. They specifically need to be trivially typable in exactly the way commas and newlines are. Realistically I don't t…

> how is the average person expected to type them? [modifier]-[C,D,E,F]

which modifier would that be, Alt/Ctrl or Cmd? Also there is a chance that this combo will not work on some platforms, some may already be in use as a hotkey.

Re: Time to retire the CSV?

#534

Post by [deleted] »

RodgerTheGreat wrote: post #470
dheera wrote: post #386

Earlier quoted context omitted.

CSV is still easier to parse because the C++ dudes still refuse to implement some kind of nice operator-overloaded interface like #include std::json myjson("{\"someArray\": [1,2,3,4,{\"a\": \"b\"}]}"); std::cout and the result is we have 50 different rogue JSON libraries instead of an STL solution. Until the STL folks wake up, boost::split can deal with the CSV.

A string split function is a poor choice if there's any possibility the CSV file contains quoted fields. Robust handling of both CSV and JSON requires a parser. In my experience, CSV can actually be trickier than JSON to parse because there are so many edge cases, alternatives, and ambiguities.

[deleted]

Re: Time to retire the CSV?

#535

Post by masklinn »

burntsushi wrote: post #417
masklinn wrote: post #249

Earlier quoted context omitted.

> There are mature CSV libraries for most major languages that handle 99% of the problems of CSV. They really don't. In fact I'd go further and confidently state that they really can't , because tons of mis-parsed CSVs are heuristic judgement values, and those tools don't really have the ability to make those calls. I've never seen a "mature CSV library for most major language" which'd guess encoding, separators, quo…

No, they really do. But 99% isn't 100%. Compare Python's csv parser with Go's. The former will prefer a parse for any input, and this is done in a fairly consistent way that at least gives one the opportunity to deal with "malformed" csv. The latter is a strict parser and coughs up an error upon almost any deviation from RFC 4180. I couldn't use the latter in practice because of it. It would just choke on too many cs…

> No, they really do. But 99% isn't 100%.

Neither is 0. Python’s csv library does not handle any problem, it applies the separators / quoting / escapes it was given and that’s that. It gives the developer the opportunity to handle things, but if the developer isn’t aware nothing is handled.

Re: Time to retire the CSV?

#536

Post by masklinn »

rendall wrote: post #362
turtlebits wrote: post #318

Earlier quoted context omitted.

Non standard delimiters. Escaping delimiters in fields - sometimes with a \, sometimes doubled (""), sometimes not at all. Double new lines. Poor handling from standard CSV libraries. Either unable to read or unable to create for some downstream process.

That sounds like the problem of badly formatted CSV, not a problem with CSV per se. If you stick to one delimiter, and that delimiter is a comma, and escape the delimiter in the data with double-quotes around the entry, and escape double quotes with two double-quotes, well, you have written CSV that is correct and looks correct and will be parsed correctly by literally every CSV parser.

> That sounds like the problem of badly formatted CSV

That’s what CSV is. That’s what happens when you ingest CSVs whose production you don’t control.

> If you [ignore everything people literally clamour for in these comments and praise csv for]

Yes i also like ponies.

Re: Time to retire the CSV?

#537

Post by masklinn »

ribosometronome wrote: post #360
masklinn wrote: post #202

Earlier quoted context omitted.

> If you don't open it in Excel, you can have as strict a parser as you want, just like any other format. No, you can not. Because the CSV format is so fuzzy you can very easily parse incorrectly and end up with a valid parse full of garbage. Trivially: incorrect separator, file happens to not contain that separator at all, you end up with a single column. That's a completely valid file, and might even make sense for…

Why is the solution to create an entirely new format rather than try to more rigidly enforce the a single CSV standard?

Because there is no way to perform that enforcement, becauae you say “CSV” and people understand “my old garbage” and you can’t fix that; and because millions of incorrect documents will yield a valid but nonsensical parse.

Re: Time to retire the CSV?

#538

Post by euroderf »

unnouinceput wrote: post #273
karteum wrote: post #264

Earlier quoted context omitted.

I personally believe that at least SQLite matches all those criteria : "1) A truly open format is available" : sqlite is open-source, MIT-licensed, and well specified (even though I am usually not so happy with its weak typing approach, yet in this case this precisely enables a 100% correspondance between CSV and sqlite since CSV has also no typing at all...) "2) Applications have a speed increase from using csvs" :…

sqlite files are binary files, so a non-starter. If I open them in Notepad I can't read the values like I can with CSV.

How about an app that opens an SQLite DB so that strings and numbers can easily be edited, and then on file save, SQLite's internal pointers and markers are updated ? Maybe an Emacs mode ?

Re: Time to retire the CSV?

#539

Post by unnah »

hermitcrab wrote: post #342
gsich wrote: post #158

Earlier quoted context omitted.

If you interpret "CSV" as purely comma seperated values then maybe. But in my bubble "CSV" means textfiles that are separated by some separator. Be it tabs, spaces, commas, or any other ASCII character. Some are more usable then others, if you have commas in your data then use tabs. If you have tabs use Form Feed or Record Separator or vertical tabs ... and so on. Of course this is not always applicable, since you so…

It seems quite common in some European countries to use semi-colons as the delimiter instead of commas (because they use commas as the decimal separator?), adding a new level of fun to parsing. In Easy Data Transform we count the number of commas, semi-colons, tabs in the file to make an educated guess as the delimiter (which the user can override).

Semicolon-separated "CSV" is a quirk of Microsoft Excel, which they have never fixed. I figure Microsoft would prefer that people use XLS instead of CSV.

Re: Time to retire the CSV?

#540

Post by mavhc »

sli wrote: post #459
hermitcrab wrote: post #408

Earlier quoted context omitted.

Maybe. It is a bit of a chicken-and-egg problem though. They aren't likely to become popular until editors support them. Also, the fact that they have been around for years and never taken off implies that they probably never will.

More generally than editor support, how is the average person expected to type them? I could easily add them to my keyboard, but if "editor support" turned out to mean "Edit > Insert > Record Separator Character" and I was now bound only to editors that support this, absolutely nobody is going to bother. They specifically need to be trivially typable in exactly the way commas and newlines are. Realistically I don't t…

Commas and Newlines are only trivial if your data contains no commas or newlines.

Who's writing CSV files by hand though?

I'd expect some kind of CSV mode in my text editor where Tab and Newlines are turned into keys to insert those characters

Post reply on HN