Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Time to retire the CSV?

bitsondisk.com

Post reply on HN
Print view View original

471–480 of 594 posts

Re: Time to retire the CSV?

#471

Post by kevin_thibedeau »

sli wrote: post #459
hermitcrab wrote: post #408

Earlier quoted context omitted.

Maybe. It is a bit of a chicken-and-egg problem though. They aren't likely to become popular until editors support them. Also, the fact that they have been around for years and never taken off implies that they probably never will.

More generally than editor support, how is the average person expected to type them? I could easily add them to my keyboard, but if "editor support" turned out to mean "Edit > Insert > Record Separator Character" and I was now bound only to editors that support this, absolutely nobody is going to bother. They specifically need to be trivially typable in exactly the way commas and newlines are. Realistically I don't t…

> how is the average person expected to type them?

[modifier]-[C,D,E,F]

Re: Time to retire the CSV?

#472

Post by ArsenArsen »

AFAICT, XML fulfils all of the requirements of this.

It is self describing, rigorously defined for the machine, highly flexible, highly extensible, human readable, text-based and an open format, and it is only mentioned as a storage format in the post, not even nearly doing it justice!

XML is complex, but there's already very established libraries for handling it, so that shouldn't be an issue, and that's the only drawback XML has.

Hell, even alternatives like JSON/YAML + JSONSchema exist, still providing incredibly rigorous validation, but staying human readable and ubiquitous.

P.S. I think a bigger failure here is trying to generalize the concept of spreadsheets, rather than using a common encoding (XML) for domain-specific data formatting.

Re: Time to retire the CSV?

#473

Post by selfhoster11 »

ArsenArsen wrote: post #472

AFAICT, XML fulfils all of the requirements of this. It is self describing, rigorously defined for the machine, highly flexible, highly extensible, human readable, text-based and an open format, and it is only mentioned as a storage format in the post, not even nearly doing it justice! XML is complex, but there's already very established libraries for handling it, so that shouldn't be an issue, and that's the only dr…

Most XML is not human-readable. There's just too much line noise, and most of what's emitted has a weird schema that's difficult to parse using Human Brain 1.0.

Re: Time to retire the CSV?

#474

Post by theamk »

masklinn wrote: post #94
Sanzig wrote: post #79

Earlier quoted context omitted.

Define "garbage." If I know what my data looks like, I can anticipate the edge cases ahead of time. Plenty of CSV exports work this way, they don't need to be general if the schema is already imposed by the system. Have you ever worked in embedded systems? Writing XML files and then zipping them on a platform with 32 kilobytes of RAM would be hell. CSV is easy, I can write the file a line at a time through a lightwei…

> Define "garbage." Incorrect encoding, incorrect separators (record and field both), incorrect escaping / quoting, etc… > If I know what my data looks like If you control the entirety of the pipeline, the format you're using is basically irrelevant. You can pick whatever you want and call it however you want. > Have you ever worked in embedded systems? Writing XML files and then zipping them on a platform with 32 ki…

> If you control the entirety of the pipeline, the format you're using is basically irrelevant.

I think you are missing the point -- you only need to know about generator to know about format.

Since the parent poster was talking embedded, here is one example: a data logger with tiny embedded records tuples: (elapsed-time, voltage, current). You need this to be readable in the widest variety of programs possible. What format do you use?

I think the answer is pretty clear: CSV. It is compatible with every programming language and spreadsheet out there, and in a pinch, you can even open it in text editor and manually examine the data.

Using something like XLSX here would be total craziness: it will make code significantly bigger, and it will severely decrease compatibility.

Re: Time to retire the CSV?

#475

Post by selfhoster11 »

asdff wrote: post #284
turtlebits wrote: post #119

Earlier quoted context omitted.

You can write what "looks" like CSV to you, but there are no guarantees it will import correctly. The problem is 10x worse when you get CSV from one source and rely on another process to load it. I fought this problem for several days going from NetSuite to Snowflake via CSV.

Can you give an example? The rules for CSV files are so simple I'm struggling to imagine a case where something looks correct but in fact isn't correct.

Lots of edge cases that aren't always handled to spec on both sides of the import/export. That's my experience, at least.

Re: Time to retire the CSV?

#476

Post by HanaShiratori »

postalrat wrote: post #406
munk-a wrote: post #367

Earlier quoted context omitted.

There is a definite demand for some import format that you can trivially edit - CSV excels at this right now and JSON isn't that bad. Binary dump formats definitely do have a time and a place but there is also a separate need for trivially human readable formats.

I guess my point is that CSVs or any text files aren't really "human readable". You still need some application top view them.

So what's the alternative? Good old pen and paper?

Re: Time to retire the CSV?

#477

Post by mr_toad »

ArsenArsen wrote: post #472

AFAICT, XML fulfils all of the requirements of this. It is self describing, rigorously defined for the machine, highly flexible, highly extensible, human readable, text-based and an open format, and it is only mentioned as a storage format in the post, not even nearly doing it justice! XML is complex, but there's already very established libraries for handling it, so that shouldn't be an issue, and that's the only dr…

Too easy to produce bloated files. People do dumb things like put field attributes and metadata on every row of the output, so you end up with files that can be several times the size of the raw data.

Parsing times are often horrible.

There’s no standard for tabular data. You invariably need some overly complicated XML map, because people can’t resist the temptation to over-engineer.

Re: Time to retire the CSV?

#478

Post by robbmorganf »

I love CSV. Only change I want is some way to indicate a comment. Then I can write a human readable description of what the data means (more than just column names, but how it is calculated). It would be cool if the comment mechanism also could have a machine readable metadata system.

Re: Time to retire the CSV?

#479

Post by sgolestane »

7thaccount wrote: post #366
Sunspark wrote: post #23

I don't agree with giving up csvs until the following conditions are met: 1) A truly open format is available and accessible. Csvs are textfiles. There is no system around that cannot open a textfile. If the format is binary or requires patents or whatever, then it's a non-starter. 2) Applications have a speed increase from using csvs. To wit, I loved csvs because often they finish preparing much faster than a "forma…

CSV is far from perfect, but it's nice that I can easily work with them without needing any libraries. All I need is file I/O and the ability to split strings. It doesn't get much simpler. I'll admit though that "import JSON" and then being able to essentially convert the entire file into a dictionary is nice if the data has more structure to it.

Nice as long as JSON is valid and not too big.

Re: Time to retire the CSV?

#480

Post by jrumbut »

7thaccount wrote: post #366
Sunspark wrote: post #23

I don't agree with giving up csvs until the following conditions are met: 1) A truly open format is available and accessible. Csvs are textfiles. There is no system around that cannot open a textfile. If the format is binary or requires patents or whatever, then it's a non-starter. 2) Applications have a speed increase from using csvs. To wit, I loved csvs because often they finish preparing much faster than a "forma…

CSV is far from perfect, but it's nice that I can easily work with them without needing any libraries. All I need is file I/O and the ability to split strings. It doesn't get much simpler. I'll admit though that "import JSON" and then being able to essentially convert the entire file into a dictionary is nice if the data has more structure to it.

The real advantage of CSV, in my mind, is that if the CSV is valid and normal then it's going to be a rectangular dataset (ignoring semantics within the dataset).

If I import JSON data I have no idea what shape the result will be in, and it requires a separate standard to let me know about columns and rows and validation can get complicated.

Post reply on HN