Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Time to retire the CSV?

bitsondisk.com

Post reply on HN
Print view View original

581–590 of 594 posts

Re: Time to retire the CSV?

#581

Post by cbsmith »

amyjess wrote: post #573
cbsmith wrote: post #502

Earlier quoted context omitted.

I'm not sure what you mean by Avro records "must" be ordered. If you mean that the serialization format specifies the ordering of the fields, then yes, that is true, but that's an advantage in terms of compactness and processing efficiency ( https://avro.apache.org/docs/current/spec.html#order ). If you don't like it though, there are other formats like protobuf and thrift that have no such requirement, at the cost o…

The problem is that fields aren't ordered in the JSON spec, so fields being ordered in Avro's dialect of JSON automatically makes it non-standard JSON which makes it difficult to use standard tooling with Avro JSON.

OIC. Yeah, in general the JSON representation is really just for cases where some tool can't handle the binary. Think of it as an escape hatch more than a core feature.

If you're using Avro as a way to do JSON, that's definitely a bad choice. That's why I found the notion of Avro as a way to work with JSON documents as really... odd. (https://news.ycombinator.com/item?id=28222491) It's a decent way to manage data transfers (and I continue to argue a much better one that CSV or JSON), but as a "lossless alternate encoding of what's canonically a JSON document", it's a bad fit.

Re: Time to retire the CSV?

#582

Post by pwinnski »

AussieWog93 wrote: post #404
pwinnski wrote: post #215

This reads like a joke. If you think you can do better than CSV, let's see your proposal. Hint: you probably can't, and if you could, you probably couldn't get Excel to export it, so you still probably can't. "The status quo is bad, more recent popular formats aren't good enough either, but I don't actually have a specific proposal that's better than all of the above" is a lot faster to read than that article, and sa…

>If you think you can do better than CSV, let's see your proposal. I've got one! It's basically the same as regular CSV, but everything is UTF-8, the columns and lines are delineated by dedicated UTF-8 "delineator" codepoints (if they aren't defined in the spec, find reasonable surrogates and use them), and therefore nothing ever needs to be escaped. More human readable than regular CSV, less prone to error and just…

So... I can't just type up a file in this new format in a text editor, since the dedicated low-byte delineator characters don't have corresponding keys on my keyboard. That makes it not quite as good as CSV for at least one use case, and certainly much less flexible.

Replacing flexible standards with more-rigid standards is always really hard.

Re: Time to retire the CSV?

#583

Post by pwinnski »

proverbialbunny wrote: post #396
pwinnski wrote: post #215

This reads like a joke. If you think you can do better than CSV, let's see your proposal. Hint: you probably can't, and if you could, you probably couldn't get Excel to export it, so you still probably can't. "The status quo is bad, more recent popular formats aren't good enough either, but I don't actually have a specific proposal that's better than all of the above" is a lot faster to read than that article, and sa…

I wouldn't mind an updated standard to CSV that has the type information in the first line with the column labels. I feel like this is all that CSV is really missing. As a format where you can easily see and interact with the data in plain text if you need to nothing really beats CSV but I believe it could be made better, including strengthening standardization. Eg, a CSV standards version header could be put at the…

Amusingly, your sibling comment had an alternate proposal that included two features, but did not in any way address the one thing you suggest is missing.

Replacing a flexible standard with a more-rigid standard is really, really hard.

Re: Time to retire the CSV?

#584

Post by Dylan16807 »

edmundsauto wrote: post #376
masklinn wrote: post #234

Earlier quoted context omitted.

> How is a csv not the most accurate representation of the data? If you trust the other agent encoded it properly in the db, then sure. The idea that a CSV would be more likely to be correctly encoded than a DB is hilarious, thanks for the laugh. But that you were confident enough to seriously put it in writing shows how little experience you have with CSV.

Yep, you got me. I'm actually a trapeze artist moonlighting on these forums. A CSV file represents the exact digits that are to be stored. You have unlimited precision. You could even store irrational numbers, equations, or mix data types in a column. OTOH, you have to make sure the delimiting character is not present in the data - that can be pretty easy, if you use the field-separator character in ASCII, or even ju…

Excel isn't a database.

Re: Time to retire the CSV?

#585

Post by Dylan16807 »

edmundsauto wrote: post #378
X6S1x6Okd1st wrote: post #235

Earlier quoted context omitted.

There's plenty of CSVs that have been produced or will be parsed by for line in input: ','.join(line) It's not exactly a problem with "CSV" specifically, but the environment in which it exists.

True, but if you don't trust the code generating the data, then any and every data format is suspect.

It's a thousand times easier to make a malformed "working" CSV than to make a malformed "working" sqlite file.

Re: Time to retire the CSV?

#586

Post by Dylan16807 »

rendall wrote: post #551
masklinn wrote: post #536

Earlier quoted context omitted.

> That sounds like the problem of badly formatted CSV That’s what CSV is. That’s what happens when you ingest CSVs whose production you don’t control. > If you [ignore everything people literally clamour for in these comments and praise csv for] Yes i also like ponies.

> "That’s what CSV is." That's really not a serious argument against CSV. Since you paraphrase in a silly way, I can do it too! Your "argument" is "Badly formatted files exist, therefore CSV bad". Everyone "against CSV" seems to be arguing against badly formatted CSV, and leaping to the conclusion that "CSV is just bad" without much more to say about it. I'm sorry that badly formatted CSV gave you a bad time, but the…

> Your "argument" is "Badly formatted files exist, therefore CSV bad".

The argument is actually that the badly formatted CSV files have taken over, therefore CSV is bad. You can't reject them, so your import becomes unreliable.

Re: Time to retire the CSV?

#587

Post by joquarky »

hermitcrab wrote: post #545
Fwirt wrote: post #542

Earlier quoted context omitted.

The other advantage not being discussed here is that once you have a csv parser, it can be trivially generalized to use any character as a delimiter, and you can usually find a character that isn't found in the data you're working with unless you're working with binary data. I work on a legacy system where passing delimited files around is usually the integration method of the day, and if you can't delimit with comma…

>it can be trivially generalized to use any character as a delimiter But then you have the additional problem of trying to work out what the delimiter is. You can take an educated guess, but this isn't going to be 100% reliable.

Make it a standard to always put the delimiter as the last character on the first line.

Re: Time to retire the CSV?

#588

Post by proverbialbunny »

pwinnski wrote: post #583
proverbialbunny wrote: post #396

Earlier quoted context omitted.

I wouldn't mind an updated standard to CSV that has the type information in the first line with the column labels. I feel like this is all that CSV is really missing. As a format where you can easily see and interact with the data in plain text if you need to nothing really beats CSV but I believe it could be made better, including strengthening standardization. Eg, a CSV standards version header could be put at the…

Amusingly, your sibling comment had an alternate proposal that included two features, but did not in any way address the one thing you suggest is missing. Replacing a flexible standard with a more-rigid standard is really, really hard.

The current standard already accepts a comment line at the top, so in this case technologically it is easy, far easier than most problems. However, getting a consensus from a committee with bike shedding and all that fun (horrible) stuff is very hard.

Re: Time to retire the CSV?

#589

Post by CRConrad »

divbzero wrote: post #373
hermitcrab wrote: post #332

Earlier quoted context omitted.

>the fact that both of its separators (newlines and commas) can appear as-is inside column values, with a different meaning, if those column-values are quoted, means that there's no way to parallelize CSV processing, because there's no way to read-ahead and "chunk" a CSV purely lexically Yes, this is a major pain. It can be avoided by using Tab separated value (TSV) files, which don't use escaping. But then you can't…

ASCII has special delimiters 0x1E Record Separator and 0x1F Unit Separator to avoid conflicting with values, but they have never gained widespread adoption.

> ASCII has special delimiters 0x1E Record Separator and 0x1F Unit Separator

And since ASCII has them, Unicode has them too.

Re: Time to retire the CSV?

#590

Post by CRConrad »

atatatat wrote: post #562
creata wrote: post #531

Earlier quoted context omitted.

In some incredibly pedantic sense, sure. But the difference is that every computer comes with what a human needs to read a CSV file. That's not true of an SQLite database.

Could it be?

Sure, could be. In about the same sense that it could be that every computer comes with a free and open source operating system. Could be; it's not theoretically impossible -- but...
Post reply on HN