Live data from Hacker News

Time to retire the CSV?

bitsondisk.com

451–460 of 594 posts

Re: Time to retire the CSV?

#451
Yeah, that's a dumb idea.

Sometimes -- frequently, even -- you have to go to some least-common-denominator format to move data around. CSV is often a candidate.

When I was younger I thought we'd eventually abandon "primitive" formats like this, but in my middle years I realize their extreme utility. CSV absolutely has a place, and absolutely does a job that no other format does as quickly or as easily.

Re: Time to retire the CSV?

#452
never gonna happen. company i work for has csv files from tests on products they conducted almost two decades ago. they went to a proprietary solution for a few years and that company went bankrupt and we can no longer open certain files because the license was restricted to Sun servers. we've learned the hard way that open systems even if sub optimal are usually the right choice especially when you're trying to store/archive data.

we've resorted to csv files + readme + json files in git for version control.

Re: Time to retire the CSV?

#453
I might have a different perspective than most, but I think csv as a raw access method is far better than any structured format.

The contract on what is transfered or how it's mapped in your csv is developer to developer and not in code.

This is a problem and also a benefit.

As long as both systems are designed to read and write the csv correctly(there in also the problem of verification), csv works flawlessly and retains its simplicity.

Re: Time to retire the CSV?

#454

I don't agree with giving up csvs until the following conditions are met: 1) A truly open format is available and accessible. Csvs are textfiles. There is no system around that cannot open a textfile. If the format is binary or requires patents or whatever, then it's a non-starter. 2) Applications have a speed increase from using csvs. To wit, I loved csvs because often they finish preparing much faster than a "forma…

CSV is far from perfect, but it's nice that I can easily work with them without needing any libraries. All I need is file I/O and the ability to split strings. It doesn't get much simpler. I'll admit though that "import JSON" and then being able to essentially convert the entire file into a dictionary is nice if the data has more structure to it.

It's funny to complain about CSV when JSON is also a minefield: http://seriot.ch/parsing_json.php

CSV is fine .. usually

Re: Time to retire the CSV?

#455
post #438

Earlier quoted context omitted.

CSV is far from perfect, but it's nice that I can easily work with them without needing any libraries. All I need is file I/O and the ability to split strings. It doesn't get much simpler. I'll admit though that "import JSON" and then being able to essentially convert the entire file into a dictionary is nice if the data has more structure to it.

> All I need is file I/O and the ability to split strings. ...until there is a newline inside a field. The moronic quoting mechanism of CSV is one half of the problem; people like you, who try to parse it by "just splitting strings" is the other half. The third half is that it's locale dependent and after 30+ years, people still don't use Unicode.

Never write your own parser, especially as just string.split(), and when possible don't use C(omma)SV formats, but C(haracter)SV, aka DSV: https://en.wikipedia.org/wiki/Delimiter-separated_values

There are non-printable, non-typable characters specifically defined as separators (ASCII 28-31) with UTF equivalents.

Re: Time to retire the CSV?

#457
The only reason CSV is hard to parse is that text editors made it hard to type ANSI 29, 30, and 31: the group, record, and field separators. So folks had to resort to comma or pipe and newline.

If you use the right characters built specifically for this purpose, then the problems go away.

Re: Time to retire the CSV?

#458

> It's Time to Retire the CSV > This column obviously contains dates, but which dates? Most of the world It's time to retire local formats and always write YYYY-MM-DD (which is both the international and the Swedish standard, and the most convenient for parsing and sorting). > A third major piece of metadata missing from CSVs is information about the file’s character encoding. It's bloody the time to retire all the c…

I had a great one today. I got a load file in (e-discovery world) that had my sent/rev'd/sort dates in "mm.dd.yyyy hh:mm" while my created and modified dates were "dd.mm.yyyy hh:mm". I can't fathom what piece of software sent that to me, and it wasn't rocket science to fix in Excel. I'm all for standardizing date format but with the # of applications that will only spit out data in one particular format I don't see it ever happening.

Re: Time to retire the CSV?

#459
post #398

Earlier quoted context omitted.

If they were popular they'd be supported in your editor though.

Maybe. It is a bit of a chicken-and-egg problem though. They aren't likely to become popular until editors support them. Also, the fact that they have been around for years and never taken off implies that they probably never will.

More generally than editor support, how is the average person expected to type them? I could easily add them to my keyboard, but if "editor support" turned out to mean "Edit > Insert > Record Separator Character" and I was now bound only to editors that support this, absolutely nobody is going to bother. They specifically need to be trivially typable in exactly the way commas and newlines are.

Realistically I don't think anyone (at least on HN) would presume that editor support would be so weirdly obtuse, but it's not exactly well-defined here, either.

I'm mostly just thinking out loud, here, not really critiquing anyone.

Re: Time to retire the CSV?

#460

Earlier quoted context omitted.

Strong agree. TSV is better than CSV always (unless you need to interface with an external system that doesn't accept it). For those not aware, TSV and CSV differ by more than just the delimiting character. TSV has a dead-simple specification: https://www.iana.org/assignments/media-types/text/tab-separa... . CSV does not have a standard spec and implementations differ quite a bit, but often in subtle ways.

There's no difference between TSV and CSV but the separator, most good libraries allow you to use any character you wish as the separator. Here's the RFC for CSV - https://datatracker.ietf.org/doc/html/rfc4180

The other key difference is instead of having complex quoting rules, TSV just disallows the tab character in the data.
Post reply on HN