Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

CSVs Are Kinda Bad. DSVs Are Kinda Good

matthodges.com

Post reply on HN
Print view View original

121–130 of 133 posts

Re: CSVs Are Kinda Bad. DSVs Are Kinda Good

#121

Post by usrbinbash »

geoelectric wrote: post #114
usrbinbash wrote: post #67

Earlier quoted context omitted.

Oh, I am nothing but liberal when it comes to CSV: Clients get the liberty to either have their requests processed, or get a 400 BAD REQUEST And yes, I am aware that the standard says this. My counter question to that is: How much client-liberty do I have to accept? Where do I draw the line? How much is too much liberty? And the answer is: there is no answer. Wherever any system draws that line, it's an arbitrary dec…

I think the suggestion reflected a deep understanding that transitioning from decades of wild-west to standardized in the smooth fashion most likely to succeed would require that strategy. If you don’t accept whatever some org’s data is encoded with, they won’t consider it a win for standards, or swap out whatever is producing that data for something more compliant. They’ll consider it a bug, and probably use some ot…

> On the other hand, if you can be flexible enough to allow quirks on import while not perpetuating them on export, eventually you and other software built with the same philosophy standardize the field.

How? The only thing I can see happening is perpetuation of sloppy use of standards. "Why, why should I change my |-deliminated CSV dialect that requires a double-semicolon at the end of each row, which is arbitrarily denoted by either \n or \r or \n\r when all those programmers will accomodate me, no matter how little sense it makes to do so?

> I do think there’s a point where things are standardized enough that you can safely stop doing that

I agree. And that point was when someone sat down, and penned RFC-4180

Everything after that point, has to justify why it isn't RFC compliant, not the other way around.

Re: CSVs Are Kinda Bad. DSVs Are Kinda Good

#122

Post by tpoacher »

usrbinbash wrote: post #17

> CSVs are kinda bad. Not really. What's bad is when people keep insisting on coming up with new and amazing CSV dialects. https://www.ietf.org/rfc/rfc4180.txt is very clear about what CSV files are supposed to look like, and the fact that people keep ignoring this for whatever reason, is not the formats problem. And no, "using another format" is not a solution to this. Because: I can just invent a new DSV dialect. O…

you mean to say that vim can't handle simple character substitution? /s

Re: CSVs Are Kinda Bad. DSVs Are Kinda Good

#123

Post by tpoacher »

People here complaining this guy is suggesting a "new" standard: it's ASCII. It is already a standard, and probably a lot more sensible than others that followed.

I too have wondered why the hell aren't we using those special characters already ever since I discovered their existence

Re: CSVs Are Kinda Bad. DSVs Are Kinda Good

#124

Post by tpoacher »

wmal wrote: post #7

The author seems to ignore the fact that CSV got so popular because it is human readable. If anyone wanted a binary format there’s plenty of them - most better than this DSV. Also, I’m on a mobile right now, so can’t verify that, but it seems the format is flawed. The reader decodes UTF8 strings after splitting the binary buffer by the delimiter, but I believe the delimiter may be a part of a UTF8 character. Edit: ju…

CSV's aren't really readable either though. They're "inspectable", but that's different. So if you want to read them you'll need to either use specific software, or do some preprocessing to align things properly etc ... in which case the extra step of performing a file-wide substitution of the record separator with newlines and unit separator with tabs or sth, isn't a much worse problem.

Re: CSVs Are Kinda Bad. DSVs Are Kinda Good

#125

Post by usrbinbash »

brunokim wrote: post #104
usrbinbash wrote: post #17

> CSVs are kinda bad. Not really. What's bad is when people keep insisting on coming up with new and amazing CSV dialects. https://www.ietf.org/rfc/rfc4180.txt is very clear about what CSV files are supposed to look like, and the fact that people keep ignoring this for whatever reason, is not the formats problem. And no, "using another format" is not a solution to this. Because: I can just invent a new DSV dialect. O…

"Broken" is a sliding scale, and it's unfeasible to refuse engaging at all times. If you are a multi-billion dollar company creating a new integration, you can demand that your small supplier provide an RFC-4180 compliant file, and even refuse to process it if its schema or encoding is not conformant. If you are the small supplier of a multi-billion dollar company, you will absolutely process whatever it is that they…

I agree up to a point. It is a kind of tug-o-war, and yes, the weight of each side plays an important role there.

Nevertheless, even in projects where my services are talking to something that's bigger, I will, at the very least ask "why cant it be RFC compliant? is there a reason?". And without blowing my own horn overly much, but quite a few systems larger than mine have changed because someone asked that question.

Re: CSVs Are Kinda Bad. DSVs Are Kinda Good

#126

Post by hyperman1 »

disgruntledphd2 wrote: post #80
SammyStacks wrote: post #51

Earlier quoted context omitted.

From a pragmatic viewpoint, the CSVs that I get from finance (usually saved as .xlsx) have the same issues for parsing the data as a CSV. But since the issues are consistent, I can automate conversion from .xlsx to CSV, then process the CSV using awk to eliminate errors in further parsing the CSV (for import, analysis, etc.). Sure, I'm essentially parsing the CSV twice but, because the parsing issues are consistent,…

As long as a human didn't generate the file, all things can be automated. However, if you ever have the misfortune of dealing with human generated files (particularly Excels) then you will suffer much pain and loss. I once had to deal with a "CSV" which had not one, not two but 6(!) distinct date formats in the same file. Life as a data scientist kinda sucks sometimes :shrug:.

Before 2010 and UTF-8 everywhere , I regularly had the misfortune of dealing with multi encoding CSVs. Someone got CSVs from multiple sources and catted them together. One source uses ISO 8859-1, another -15, another UTF-8, sometimes a greek or russian or even ebcdic was in there. Fun trying to guess where one stopped and the other begun . Of course, none of them were consistent crlf or escape wise.

Re: CSVs Are Kinda Bad. DSVs Are Kinda Good

#128

Post by funcDropShadow »

Freak_NL wrote: post #42
throw0101d wrote: post #24

Earlier quoted context omitted.

[flagged]

Importing data generated from Excel: don't, or force RFC 4180-compliant CSV data. Exporting data into Excel: provide RFC 4180-compliant CSV data, or just generate minimal XLSX files. Most Excel users generally don't export to CSV (or practice any data sanity conventions); they seem to believe XLSX is a perfectly fine data exchange format for automated use. (“Oh the import broke? I just added an empty row above the he…

Do you have some recommendations or references about getting data properly out of Excel. I usually avoid using Excel alltogether if possible. But, obviously that is not always an option.

Re: CSVs Are Kinda Bad. DSVs Are Kinda Good

#129

Post by teddyh »

tttp wrote: post #118
usrbinbash wrote: post #70

Earlier quoted context omitted.

It doesn't matter when it came out, it doesn't matter that it it descriptive. It is the standard, period. Yes, CSV is much, much older. In fact it predates personal computers. And it went through changes. Again: None of that matters. We have a standard, we should use the standard, and systems should demand the standard. Standards are meant to ensure minimal-friction interoperability. If systems don't enforce standard…

"This memo provides information for the Internet community. It does not specify an Internet standard of any kind."

Note the qualifier: “not an Internet standard” (my emphasis).

Re: CSVs Are Kinda Bad. DSVs Are Kinda Good

#130

Post by usrbinbash »

teddyh wrote: post #129
tttp wrote: post #118

Earlier quoted context omitted.

"This memo provides information for the Internet community. It does not specify an Internet standard of any kind."

Note the qualifier: “not an Internet standard” (my emphasis).

And again: None of that matters. I am not talking about formalities here, I am talking about technical realities.

Whether it is formally called a standard or no doesn't change the fact that this is the document everyone points at when determining what CSV is and is supposed to look like. So it is de-facto a standard. Call it a "quasi standard" if that makes you happy.

Post reply on HN